Nemo Curator

orchestra-research/ai-research-skills/05-data-processing/nemo-curator

by orchestra-research773a52944ba4MIT13K starsListed Oct 8, 2026Updated Oct 8, 2026Repository updated 3 months ago

GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.

Instructions onlyData & AnalyticsAI & Agents
AI-generated overview

Curates and cleans LLM training data with GPU-accelerated filtering, deduplication, PII redaction and multimodal support.

What it does
Guides the use of NVIDIA NeMo Curator to build data curation pipelines for LLM training data. It covers quality filtering with heuristics, exact, fuzzy and semantic deduplication, PII redaction, classifier filtering, and image, video and audio curation. It produces cleaned datasets written to Parquet or JSONL and includes performance and cost comparisons for GPU versus CPU processing.
When to use it
Use it when preparing training data from web scrapes such as Common Crawl, deduplicating large text corpora, or filtering low-quality, toxic or NSFW content. It also fits curating multimodal datasets and scaling curation across a GPU cluster.
Requirements
Requires the nemo-curator package with cudf, dask and RAPIDS, installed via pip with a CUDA or CPU extra. GPU acceleration needs CUDA-capable hardware and a Dask GPU cluster; some classifiers and embedding models are downloaded from model hubs, so network access is needed. The skill ships no scripts, only reference documents.

NeMo Curator - GPU-Accelerated Data Curation

NVIDIA's toolkit for preparing high-quality training data for LLMs.

When to use NeMo Curator

Use NeMo Curator when:

  • Preparing LLM training data from web scrapes (Common Crawl)
  • Need fast deduplication (16× faster than CPU)
  • Curating multi-modal datasets (text, images, video, audio)
  • Filtering low-quality or toxic content
  • Scaling data processing across GPU cluster

Performance:

  • 16× faster fuzzy deduplication (8TB RedPajama v2)
  • 40% lower TCO vs CPU alternatives
  • Near-linear scaling across GPU nodes

Use alternatives instead:

  • datatrove: CPU-based, open-source data processing
  • dolma: Allen AI's data toolkit
  • Ray Data: General ML data processing (no curation focus)

Quick start

Installation

bash
# Text curation (CUDA 12)uv pip install "nemo-curator[text_cuda12]"
# All modalitiesuv pip install "nemo-curator[all_cuda12]"
# CPU-only (slower)uv pip install "nemo-curator[cpu]"

Basic text curation pipeline

python
from nemo_curator import ScoreFilter, Modifyfrom nemo_curator.datasets import DocumentDatasetimport pandas as pd
# Load datadf = pd.DataFrame({"text": ["Good document", "Bad doc", "Excellent text"]})dataset = DocumentDataset(df)
# Quality filteringdef quality_score(doc):    return len(doc["text"].split()) > 5  # Filter short docs
filtered = ScoreFilter(quality_score)(dataset)
# Deduplicationfrom nemo_curator.modules import ExactDuplicatesdeduped = ExactDuplicates()(filtered)
# Savededuped.to_parquet("curated_data/")

Data curation pipeline

Stage 1: Quality filtering

python
from nemo_curator.filters import (    WordCountFilter,    RepeatedLinesFilter,    UrlRatioFilter,    NonAlphaNumericFilter)
# Apply 30+ heuristic filtersfrom nemo_curator import ScoreFilter
# Word count filterdataset = dataset.filter(WordCountFilter(min_words=50, max_words=100000))
# Remove repetitive contentdataset = dataset.filter(RepeatedLinesFilter(max_repeated_line_fraction=0.3))
# URL ratio filterdataset = dataset.filter(UrlRatioFilter(max_url_ratio=0.2))

Stage 2: Deduplication

Exact deduplication:

python
from nemo_curator.modules import ExactDuplicates
# Remove exact duplicatesdeduped = ExactDuplicates(id_field="id", text_field="text")(dataset)

Fuzzy deduplication (16× faster on GPU):

python
from nemo_curator.modules import FuzzyDuplicates
# MinHash + LSH deduplicationfuzzy_dedup = FuzzyDuplicates(    id_field="id",    text_field="text",    num_hashes=260,      # MinHash parameters    num_buckets=20,    hash_method="md5")
deduped = fuzzy_dedup(dataset)

Semantic deduplication:

python
from nemo_curator.modules import SemanticDuplicates
# Embedding-based deduplicationsemantic_dedup = SemanticDuplicates(    id_field="id",    text_field="text",    embedding_model="sentence-transformers/all-MiniLM-L6-v2",    threshold=0.8  # Cosine similarity threshold)
deduped = semantic_dedup(dataset)

Stage 3: PII redaction

python
from nemo_curator.modules import Modifyfrom nemo_curator.modifiers import PIIRedactor
# Redact personally identifiable informationpii_redactor = PIIRedactor(    supported_entities=["EMAIL_ADDRESS", "PHONE_NUMBER", "PERSON", "LOCATION"],    anonymize_action="replace"  # or "redact")
redacted = Modify(pii_redactor)(dataset)

Stage 4: Classifier filtering

python
from nemo_curator.classifiers import QualityClassifier
# Quality classificationquality_clf = QualityClassifier(    model_path="nvidia/quality-classifier-deberta",    batch_size=256,    device="cuda")
# Filter low-quality documentshigh_quality = dataset.filter(lambda doc: quality_clf(doc["text"]) > 0.5)

GPU acceleration

GPU vs CPU performance

OperationCPU (16 cores)GPU (A100)Speedup
Fuzzy dedup (8TB)120 hours7.5 hours16×
Exact dedup (1TB)8 hours0.5 hours16×
Quality filtering2 hours0.2 hours10×

Multi-GPU scaling

python
from nemo_curator import get_clientimport dask_cuda
# Initialize GPU clusterclient = get_client(cluster_type="gpu", n_workers=8)
# Process with 8 GPUsdeduped = FuzzyDuplicates(...)(dataset)

Multi-modal curation

Image curation

python
from nemo_curator.image import (    AestheticFilter,    NSFWFilter,    CLIPEmbedder)
# Aesthetic scoringaesthetic_filter = AestheticFilter(threshold=5.0)filtered_images = aesthetic_filter(image_dataset)
# NSFW detectionnsfw_filter = NSFWFilter(threshold=0.9)safe_images = nsfw_filter(filtered_images)
# Generate CLIP embeddingsclip_embedder = CLIPEmbedder(model="openai/clip-vit-base-patch32")image_embeddings = clip_embedder(safe_images)

Video curation

python
from nemo_curator.video import (    SceneDetector,    ClipExtractor,    InternVideo2Embedder)
# Detect scenesscene_detector = SceneDetector(threshold=27.0)scenes = scene_detector(video_dataset)
# Extract clipsclip_extractor = ClipExtractor(min_duration=2.0, max_duration=10.0)clips = clip_extractor(scenes)
# Generate embeddingsvideo_embedder = InternVideo2Embedder()video_embeddings = video_embedder(clips)

Audio curation

python
from nemo_curator.audio import (    ASRInference,    WERFilter,    DurationFilter)
# ASR transcriptionasr = ASRInference(model="nvidia/stt_en_fastconformer_hybrid_large_pc")transcribed = asr(audio_dataset)
# Filter by WER (word error rate)wer_filter = WERFilter(max_wer=0.3)high_quality_audio = wer_filter(transcribed)
# Duration filteringduration_filter = DurationFilter(min_duration=1.0, max_duration=30.0)filtered_audio = duration_filter(high_quality_audio)

Common patterns

Web scrape curation (Common Crawl)

python
from nemo_curator import ScoreFilter, Modifyfrom nemo_curator.filters import *from nemo_curator.modules import *from nemo_curator.datasets import DocumentDataset
# Load Common Crawl datadataset = DocumentDataset.read_parquet("common_crawl/*.parquet")
# Pipelinepipeline = [    # 1. Quality filtering    WordCountFilter(min_words=100, max_words=50000),    RepeatedLinesFilter(max_repeated_line_fraction=0.2),    SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3),    UrlRatioFilter(max_url_ratio=0.3),
    # 2. Language filtering    LanguageIdentificationFilter(target_languages=["en"]),
    # 3. Deduplication    ExactDuplicates(id_field="id", text_field="text"),    FuzzyDuplicates(id_field="id", text_field="text", num_hashes=260),
    # 4. PII redaction    PIIRedactor(),
    # 5. NSFW filtering    NSFWClassifier(threshold=0.8)]
# Executefor stage in pipeline:    dataset = stage(dataset)
# Savedataset.to_parquet("curated_common_crawl/")

Distributed processing

python
from nemo_curator import get_clientfrom dask_cuda import LocalCUDACluster
# Multi-GPU clustercluster = LocalCUDACluster(n_workers=8)client = get_client(cluster=cluster)
# Process large datasetdataset = DocumentDataset.read_parquet("s3://large_dataset/*.parquet")deduped = FuzzyDuplicates(...)(dataset)
# Cleanupclient.close()cluster.close()

Performance benchmarks

Fuzzy deduplication (8TB RedPajama v2)

  • CPU (256 cores): 120 hours
  • GPU (8× A100): 7.5 hours
  • Speedup: 16×

Exact deduplication (1TB)

  • CPU (64 cores): 8 hours
  • GPU (4× A100): 0.5 hours
  • Speedup: 16×

Quality filtering (100GB)

  • CPU (32 cores): 2 hours
  • GPU (2× A100): 0.2 hours
  • Speedup: 10×

Cost comparison

CPU-based curation (AWS c5.18xlarge × 10):

  • Cost: $3.60/hour × 10 = $36/hour
  • Time for 8TB: 120 hours
  • Total: $4,320

GPU-based curation (AWS p4d.24xlarge × 2):

  • Cost: $32.77/hour × 2 = $65.54/hour
  • Time for 8TB: 7.5 hours
  • Total: $491.55

Savings: 89% reduction ($3,828 saved)

Supported data formats

  • Input: Parquet, JSONL, CSV
  • Output: Parquet (recommended), JSONL
  • WebDataset: TAR archives for multi-modal

Use cases

Production deployments:

  • NVIDIA used NeMo Curator to prepare Nemotron-4 training data
  • Open-source datasets curated: RedPajama v2, The Pile

References

  • Filtering Guide [blocked] - 30+ quality filters, heuristics
  • Deduplication Guide [blocked] - Exact, fuzzy, semantic methods

Resources

Source and attribution

Source:orchestra-research/ai-research-skillsin05-data-processing/nemo-curatorat commit773a529

License: MIT

Content belongs to its original authors. SourceWeft indexes it from a public repository.

Report or request removal

More from orchestra-research/ai-research-skills

Speculative Decoding

orchestra-research

Guides accelerating LLM inference with speculative decoding, Medusa multi-head prediction, and lookahead decoding.

AI & Agents13Kupdated 3 months ago

Systems Paper Writing

orchestra-research

Guides paragraph-level structure and page budgets for writing systems conference papers such as OSDI, SOSP, ASPLOS and NSDI.

Writing & Content13Kupdated 3 months ago

Moe Training

orchestra-research

Guides training and inference of Mixture of Experts models with DeepSpeed or HuggingFace, covering routing, load balancing and expert parallelism.

AI & Agents13Kupdated 3 months ago

Presenting Conference Talks

orchestra-research

Generates conference presentation slides (Beamer LaTeX PDF and editable PPTX) from a compiled paper with speaker notes and talk script. Use when preparing oral talks, spotlight presentations, or invited talks for ML and systems conferences.

Awaiting classification13Kupdated 3 months ago

Model Pruning

orchestra-research

Guides LLM pruning with Wanda, SparseGPT, magnitude and N:M methods to cut model size and speed inference.

AI & Agents13Kupdated 3 months ago

Outlines

orchestra-research

Guides use of the Outlines library for structured, schema-constrained text generation with local models.

AI & Agents13Kupdated 3 months ago

More in Data & Analytics

Swap Curve Strategy

anthropics

Featured

Analyzes interest rate swap curves, swap spreads, real rates and curve trade ideas using MCP pricing tools.

Data & Analytics39Kupdated 2 weeks ago

Unit Economics

anthropics

Featured

Analyzes unit economics for private equity targets: ARR cohorts, LTV/CAC, net retention, payback and margin waterfall.

Data & Analytics39Kupdated 2 weeks ago

Option Vol Analysis

anthropics

Featured

Analyzes option volatility by combining vol surfaces, option pricing with Greeks, and historical prices to compare implied and realized vol.

Data & Analytics39Kupdated 2 weeks ago

Macro Rates Monitor

anthropics

Featured

Builds macro and rates dashboards from MCP data on GDP, inflation, yield curves, breakevens and swaps.

Data & Analytics39Kupdated 2 weeks ago

Clean Data Xls

anthropics

Featured

Cleans messy spreadsheet data: trims whitespace, fixes casing, converts text numbers, standardizes dates, and removes duplicates.

Data & Analytics39Kupdated 2 weeks ago

Fixed Income Portfolio

anthropics

Featured

Reviews fixed income portfolios by pricing bonds, aggregating risk metrics, projecting cashflows and running rate scenarios.

Data & Analytics39Kupdated 2 weeks ago