Domino Datasets

Domino Datasets Skill

Description

This skill helps users work with Domino Datasets - high-performance, versioned filesystem storage for data science projects.

Activation

Activate this skill when users want to:

• Create or manage Domino Datasets
• Work with dataset snapshots and versioning
• Share data between projects
• Access large datasets efficiently
• Understand dataset paths and mounting

What is a Domino Dataset?

A Domino Dataset is:

• High-performance storage: Network filesystem optimized for data science
• Versioned: Create snapshots for reproducibility
• Shareable: Access across projects
• Scalable: No file size or count limits
• Persistent: Data persists across executions

Creating a Dataset

Via Domino UI

1. Navigate to your project
2. Go to Data > Domino Datasets
3. Click Create New Dataset
4. Enter:
   • Name: Dataset name (e.g., training-data)
   • Description: What the dataset contains
5. Click Create

Via Python SDK

from domino import Domino
domino = Domino("project-owner/project-name")
# Create a new datasetdataset = domino.datasets_create(    name="training-data",    description="Training data for classification model")

Dataset Paths

Dataset paths differ based on your project type. Domino has two project types with different mount structures.

DFS (Domino File System) Projects

DFS projects use /domino as the root:

/domino   |--/datasets      |--/local               <== Local datasets and snapshots         |--/clapton          <== Read-write dataset for owner and editor, read-only for reader         |--/mingus           <== Read-write dataset for owner and editor, read-only for reader         |--/snapshots        <== Snapshot folder organized by dataset            |--/clapton       <== Read-write for owner and editor, read-only for reader               |--/tag1          <== Mounted under latest tag               |--/1             <== Always mounted under the snapshot number               |--/2            |--/mingus               |--/tag2               |--/1               |--/2      |--/ella                <== Read-write shared dataset for owner and editor, Read-only for reader      |--/davis               <== Read-write shared dataset for owner and editor, Read-only for reader      |--/snapshots           <== Shared datasets snapshots organized by dataset         |--/ella             <== Read-write for owner and editor, read-only for reader            |--/tag3          <== Mounted under latest tag            |--/1             <== Always mounted under the snapshot number            |--/2         |--/davis            |--/tag4            |--/1            |--/2

Git-Based Projects

Git-based projects use /mnt as the root:

/mnt   |--/data                  <== Local datasets and snapshots     |--/clapton             <== Read-write dataset for owner and editor, read-only for reader     |--/mingus              <== Read-write dataset for owner and editor, read-only for reader     |--/snapshots           <== Snapshot folder organized by dataset        |--/clapton          <== Read-write for owner and editor, read-only for reader           |--/tag1          <== Mounted under latest tag           |--/1             <== Always mounted under the snapshot number           |--/2        |--/mingus           |--/tag2           |--/1           |--/2   |--/imported     |--/data        |--/ella             <== Read-write shared dataset for owner and editor, read-only for reader        |--/davis            <== Read-write shared dataset for owner and editor, read-only for reader        |--/snapshots        <== Shared dataset snapshots organized by dataset           |--/ella          <== Read-write for owner and editor, read-only for reader              |--/tag3       <== Mounted under latest tag              |--/1          <== Always mounted under the snapshot number              |--/2           |--/davis              |--/tag4              |--/1              |--/2

How to Identify Your Project Type

Check which paths exist in your execution:

import os
if os.path.exists("/domino/datasets"):    print("DFS Project")    dataset_root = "/domino/datasets/local"elif os.path.exists("/mnt/data"):    print("Git-Based Project")    dataset_root = "/mnt/data"

Permissions

Both project types follow the same permission model:

• Owners/Editors: Read-write access to datasets
• Readers: Read-only access

Example: Reading Data

import pandas as pd
# Git-Based Projectdf = pd.read_csv("/mnt/data/training-data/customers.csv")
# DFS Projectdf = pd.read_csv("/domino/datasets/local/training-data/customers.csv")
# List filesimport osfiles = os.listdir("/mnt/data/training-data/")  # Git-Basedfiles = os.listdir("/domino/datasets/local/training-data/")  # DFS

Uploading Data

Via Domino UI

1. Go to dataset page
2. Click Upload
3. Select files (up to 50GB or 50,000 files via UI)
4. Click Upload

Via Domino CLI (Large Uploads)

# For large uploads, use CLIdomino upload /local/path/to/data /mnt/data/training-data/

Via Code in Workspace

import shutil
# Copy from local to datasetshutil.copy("local_file.csv", "/mnt/data/training-data/")
# Write directlydf.to_csv("/mnt/data/training-data/processed.csv", index=False)

Snapshots

What is a Snapshot?

A snapshot is a read-only, immutable version of your dataset at a point in time. Use snapshots for:

• Reproducibility
• Versioning training data
• Rolling back to previous states

Create a Snapshot

# Via Python SDKsnapshot = domino.datasets_snapshot(    dataset_name="training-data",    tag="v1.0")

Or via UI:

1. Go to dataset page
2. Click Create Snapshot
3. Add optional tag (e.g., v1.0, production)

Access Snapshots

# Latest snapshotdf = pd.read_csv("/mnt/data/training-data/data.csv")
# Specific tagged snapshotdf = pd.read_csv("/mnt/data/[email protected]/data.csv")

Snapshot Limits

• Default limit: 20 snapshots per dataset
• Configurable by admins
• Oldest snapshots auto-deleted when limit reached

Tags

What are Tags?

Tags provide friendly names for snapshots:

• production: Current production data
• v1.0, v2.0: Version numbers
• 2024-01-15: Date-based tags

Move Tags

Tags can be moved to different snapshots:

# Move 'production' tag to latest snapshotdomino.datasets_tag(    dataset_name="training-data",    snapshot_id="snapshot-123",    tag="production")

Sharing Datasets

Within Organization

1. Go to dataset settings
2. Set visibility to Organization
3. Other projects can mount the dataset

Cross-Project Access

# Import dataset from another project# Configured in project settingsdf = pd.read_csv("/mnt/data/shared-dataset/data.csv")

Best Practices

1. Use Appropriate Storage

2. Organize Data

/mnt/data/my-dataset/├── raw/│   ├── customers.csv│   └── transactions.csv├── processed/│   ├── features.parquet│   └── labels.parquet└── metadata/    └── schema.json

3. Use Efficient Formats

# Parquet for tabular data (faster, smaller)df.to_parquet("/mnt/data/dataset/data.parquet")
# Feather for pandas DataFramesdf.to_feather("/mnt/data/dataset/data.feather")
# HDF5 for numerical arraysimport h5pywith h5py.File("/mnt/data/dataset/data.h5", "w") as f:    f.create_dataset("features", data=features)

4. Document Data

Include README and schema:

# Write metadatametadata = {    "created": "2024-01-15",    "source": "Customer database",    "columns": {"id": "int", "name": "string", "value": "float"}}
with open("/mnt/data/dataset/metadata.json", "w") as f:    json.dump(metadata, f)

5. Snapshot Before Changes

# Create snapshot before processingdomino.datasets_snapshot(    dataset_name="training-data",    tag="pre-processing")
# Then modify dataprocess_data()

Reading Large Datasets

Chunked Reading

# Read in chunkschunks = pd.read_csv(    "/mnt/data/dataset/large_file.csv",    chunksize=100000)
for chunk in chunks:    process(chunk)

Lazy Loading with Dask

import dask.dataframe as dd
# Read without loading into memorydf = dd.read_parquet("/mnt/data/dataset/large_data.parquet")
# Process lazilyresult = df.groupby("category").mean().compute()

Memory Mapping

import numpy as np
# Memory-map large arraysdata = np.memmap(    "/mnt/data/dataset/features.dat",    dtype='float32',    mode='r',    shape=(1000000, 100))

Troubleshooting

Dataset Not Found

• Verify dataset name is correct
• Check dataset is mounted to project
• Confirm you have access permissions

Permission Denied

• Check project role (need Contributor+)
• Verify dataset sharing settings
• Contact dataset owner

Slow Performance

• Use efficient file formats (Parquet > CSV)
• Read only needed columns
• Use chunked/lazy loading for large files

Snapshot Failed

• Check disk quota
• Verify no files are open/locked
• Check snapshot limit not reached

Documentation Reference

• Create and manage Datasets
• Version data with Snapshots
• Datasets best practices