Nemo Curator

orchestra-research/ai-research-skills/05-data-processing/nemo-curator

作者 orchestra-research773a52944ba4MIT13K 個星標收錄於 2026年10月8日更新於 2026年10月8日儲存庫3 個月前更新

GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.

僅含說明Data & AnalyticsAI & Agents
AI 產生的概覽

針對 LLM 訓練資料的整理與清理，支援 GPU 加速的過濾、去重、PII 去識別化與多模態處理。

功能
指導使用 NVIDIA NeMo Curator 建立 LLM 訓練資料整理流程。內容涵蓋啟發式品質過濾、精確、模糊與語意去重、PII 去識別化、分類器過濾，以及影像、視訊與音訊整理。產出為寫入 Parquet 或 JSONL 的清理後資料集，並提供 GPU 與 CPU 處理的效能與成本比較。
適用情境
適用於從 Common Crawl 等網頁擷取資料準備訓練集、對大規模文字語料去重，或過濾低品質、有害及 NSFW 內容。也適合整理多模態資料集，並在 GPU 叢集上擴展整理作業。
執行需求
需要 nemo-curator 套件以及 cudf、dask 與 RAPIDS，透過 pip 搭配 CUDA 或 CPU 附加項目安裝。GPU 加速需要支援 CUDA 的硬體與 Dask GPU 叢集；部分分類器與嵌入模型需從模型倉庫下載，因此需要網路存取。此技能不含指令碼，只有參考文件。

NeMo Curator - GPU-Accelerated Data Curation

NVIDIA's toolkit for preparing high-quality training data for LLMs.

When to use NeMo Curator

Use NeMo Curator when:

  • Preparing LLM training data from web scrapes (Common Crawl)
  • Need fast deduplication (16× faster than CPU)
  • Curating multi-modal datasets (text, images, video, audio)
  • Filtering low-quality or toxic content
  • Scaling data processing across GPU cluster

Performance:

  • 16× faster fuzzy deduplication (8TB RedPajama v2)
  • 40% lower TCO vs CPU alternatives
  • Near-linear scaling across GPU nodes

Use alternatives instead:

  • datatrove: CPU-based, open-source data processing
  • dolma: Allen AI's data toolkit
  • Ray Data: General ML data processing (no curation focus)

Quick start

Installation

bash
# Text curation (CUDA 12)uv pip install "nemo-curator[text_cuda12]"
# All modalitiesuv pip install "nemo-curator[all_cuda12]"
# CPU-only (slower)uv pip install "nemo-curator[cpu]"

Basic text curation pipeline

python
from nemo_curator import ScoreFilter, Modifyfrom nemo_curator.datasets import DocumentDatasetimport pandas as pd
# Load datadf = pd.DataFrame({"text": ["Good document", "Bad doc", "Excellent text"]})dataset = DocumentDataset(df)
# Quality filteringdef quality_score(doc):    return len(doc["text"].split()) > 5  # Filter short docs
filtered = ScoreFilter(quality_score)(dataset)
# Deduplicationfrom nemo_curator.modules import ExactDuplicatesdeduped = ExactDuplicates()(filtered)
# Savededuped.to_parquet("curated_data/")

Data curation pipeline

Stage 1: Quality filtering

python
from nemo_curator.filters import (    WordCountFilter,    RepeatedLinesFilter,    UrlRatioFilter,    NonAlphaNumericFilter)
# Apply 30+ heuristic filtersfrom nemo_curator import ScoreFilter
# Word count filterdataset = dataset.filter(WordCountFilter(min_words=50, max_words=100000))
# Remove repetitive contentdataset = dataset.filter(RepeatedLinesFilter(max_repeated_line_fraction=0.3))
# URL ratio filterdataset = dataset.filter(UrlRatioFilter(max_url_ratio=0.2))

Stage 2: Deduplication

Exact deduplication:

python
from nemo_curator.modules import ExactDuplicates
# Remove exact duplicatesdeduped = ExactDuplicates(id_field="id", text_field="text")(dataset)

Fuzzy deduplication (16× faster on GPU):

python
from nemo_curator.modules import FuzzyDuplicates
# MinHash + LSH deduplicationfuzzy_dedup = FuzzyDuplicates(    id_field="id",    text_field="text",    num_hashes=260,      # MinHash parameters    num_buckets=20,    hash_method="md5")
deduped = fuzzy_dedup(dataset)

Semantic deduplication:

python
from nemo_curator.modules import SemanticDuplicates
# Embedding-based deduplicationsemantic_dedup = SemanticDuplicates(    id_field="id",    text_field="text",    embedding_model="sentence-transformers/all-MiniLM-L6-v2",    threshold=0.8  # Cosine similarity threshold)
deduped = semantic_dedup(dataset)

Stage 3: PII redaction

python
from nemo_curator.modules import Modifyfrom nemo_curator.modifiers import PIIRedactor
# Redact personally identifiable informationpii_redactor = PIIRedactor(    supported_entities=["EMAIL_ADDRESS", "PHONE_NUMBER", "PERSON", "LOCATION"],    anonymize_action="replace"  # or "redact")
redacted = Modify(pii_redactor)(dataset)

Stage 4: Classifier filtering

python
from nemo_curator.classifiers import QualityClassifier
# Quality classificationquality_clf = QualityClassifier(    model_path="nvidia/quality-classifier-deberta",    batch_size=256,    device="cuda")
# Filter low-quality documentshigh_quality = dataset.filter(lambda doc: quality_clf(doc["text"]) > 0.5)

GPU acceleration

GPU vs CPU performance

OperationCPU (16 cores)GPU (A100)Speedup
Fuzzy dedup (8TB)120 hours7.5 hours16×
Exact dedup (1TB)8 hours0.5 hours16×
Quality filtering2 hours0.2 hours10×

Multi-GPU scaling

python
from nemo_curator import get_clientimport dask_cuda
# Initialize GPU clusterclient = get_client(cluster_type="gpu", n_workers=8)
# Process with 8 GPUsdeduped = FuzzyDuplicates(...)(dataset)

Multi-modal curation

Image curation

python
from nemo_curator.image import (    AestheticFilter,    NSFWFilter,    CLIPEmbedder)
# Aesthetic scoringaesthetic_filter = AestheticFilter(threshold=5.0)filtered_images = aesthetic_filter(image_dataset)
# NSFW detectionnsfw_filter = NSFWFilter(threshold=0.9)safe_images = nsfw_filter(filtered_images)
# Generate CLIP embeddingsclip_embedder = CLIPEmbedder(model="openai/clip-vit-base-patch32")image_embeddings = clip_embedder(safe_images)

Video curation

python
from nemo_curator.video import (    SceneDetector,    ClipExtractor,    InternVideo2Embedder)
# Detect scenesscene_detector = SceneDetector(threshold=27.0)scenes = scene_detector(video_dataset)
# Extract clipsclip_extractor = ClipExtractor(min_duration=2.0, max_duration=10.0)clips = clip_extractor(scenes)
# Generate embeddingsvideo_embedder = InternVideo2Embedder()video_embeddings = video_embedder(clips)

Audio curation

python
from nemo_curator.audio import (    ASRInference,    WERFilter,    DurationFilter)
# ASR transcriptionasr = ASRInference(model="nvidia/stt_en_fastconformer_hybrid_large_pc")transcribed = asr(audio_dataset)
# Filter by WER (word error rate)wer_filter = WERFilter(max_wer=0.3)high_quality_audio = wer_filter(transcribed)
# Duration filteringduration_filter = DurationFilter(min_duration=1.0, max_duration=30.0)filtered_audio = duration_filter(high_quality_audio)

Common patterns

Web scrape curation (Common Crawl)

python
from nemo_curator import ScoreFilter, Modifyfrom nemo_curator.filters import *from nemo_curator.modules import *from nemo_curator.datasets import DocumentDataset
# Load Common Crawl datadataset = DocumentDataset.read_parquet("common_crawl/*.parquet")
# Pipelinepipeline = [    # 1. Quality filtering    WordCountFilter(min_words=100, max_words=50000),    RepeatedLinesFilter(max_repeated_line_fraction=0.2),    SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3),    UrlRatioFilter(max_url_ratio=0.3),
    # 2. Language filtering    LanguageIdentificationFilter(target_languages=["en"]),
    # 3. Deduplication    ExactDuplicates(id_field="id", text_field="text"),    FuzzyDuplicates(id_field="id", text_field="text", num_hashes=260),
    # 4. PII redaction    PIIRedactor(),
    # 5. NSFW filtering    NSFWClassifier(threshold=0.8)]
# Executefor stage in pipeline:    dataset = stage(dataset)
# Savedataset.to_parquet("curated_common_crawl/")

Distributed processing

python
from nemo_curator import get_clientfrom dask_cuda import LocalCUDACluster
# Multi-GPU clustercluster = LocalCUDACluster(n_workers=8)client = get_client(cluster=cluster)
# Process large datasetdataset = DocumentDataset.read_parquet("s3://large_dataset/*.parquet")deduped = FuzzyDuplicates(...)(dataset)
# Cleanupclient.close()cluster.close()

Performance benchmarks

Fuzzy deduplication (8TB RedPajama v2)

  • CPU (256 cores): 120 hours
  • GPU (8× A100): 7.5 hours
  • Speedup: 16×

Exact deduplication (1TB)

  • CPU (64 cores): 8 hours
  • GPU (4× A100): 0.5 hours
  • Speedup: 16×

Quality filtering (100GB)

  • CPU (32 cores): 2 hours
  • GPU (2× A100): 0.2 hours
  • Speedup: 10×

Cost comparison

CPU-based curation (AWS c5.18xlarge × 10):

  • Cost: $3.60/hour × 10 = $36/hour
  • Time for 8TB: 120 hours
  • Total: $4,320

GPU-based curation (AWS p4d.24xlarge × 2):

  • Cost: $32.77/hour × 2 = $65.54/hour
  • Time for 8TB: 7.5 hours
  • Total: $491.55

Savings: 89% reduction ($3,828 saved)

Supported data formats

  • Input: Parquet, JSONL, CSV
  • Output: Parquet (recommended), JSONL
  • WebDataset: TAR archives for multi-modal

Use cases

Production deployments:

  • NVIDIA used NeMo Curator to prepare Nemotron-4 training data
  • Open-source datasets curated: RedPajama v2, The Pile

References

  • Filtering Guide [blocked] - 30+ quality filters, heuristics
  • Deduplication Guide [blocked] - Exact, fuzzy, semantic methods

Resources

來源與署名

來源：orchestra-research/ai-research-skills位於05-data-processing/nemo-curator提交773a529

授權條款： MIT

內容歸原作者所有。SourceWeft 從公開儲存庫中收錄這些內容。

檢舉或申請下架

更多來自 orchestra-research/ai-research-skills 的技能

Speculative Decoding

orchestra-research

指導使用推測解碼、Medusa 多頭預測與前瞻解碼來加速大型語言模型推論。

AI & Agents13K3 個月前更新

Systems Paper Writing

orchestra-research

為 OSDI、SOSP、ASPLOS、NSDI 等系統會議論文寫作提供段落層級結構與篇幅分配指引。

Writing & Content13K3 個月前更新

Moe Training

orchestra-research

指導使用 DeepSpeed 或 HuggingFace 訓練與推論混合專家模型，涵蓋路由、負載平衡與專家平行。

AI & Agents13K3 個月前更新

Presenting Conference Talks

orchestra-research

Generates conference presentation slides (Beamer LaTeX PDF and editable PPTX) from a compiled paper with speaker notes and talk script. Use when preparing oral talks, spotlight presentations, or invited talks for ML and systems conferences.

待分類13K3 個月前更新

Model Pruning

orchestra-research

指導使用 Wanda、SparseGPT、幅度剪枝與 N:M 方法壓縮 LLM，縮小模型體積並加速推論。

AI & Agents13K3 個月前更新

Outlines

orchestra-research

指導使用 Outlines 函式庫，透過本機模型進行受結構約束的文字生成。

AI & Agents13K3 個月前更新

更多Data & Analytics技能

Audit Xls

anthropics

精選

稽核試算表公式與財務模型，找出錯誤、硬編碼和完整性問題並產出報告。

Data & Analytics39K2 週前更新

Comps Analysis

anthropics

精選

以 Excel 試算表形式建立機構級可比公司分析，涵蓋營運指標、估值倍數與統計基準。

Data & Analytics39K2 週前更新

Audit Xls

anthropics

精選

稽核試算表公式與財務模型，找出錯誤、硬編碼和完整性問題並產出報告。

Data & Analytics39K2 週前更新

Audit Xls

anthropics

精選

稽核試算表公式與財務模型，找出錯誤、硬編碼和完整性問題並產出報告。

Data & Analytics39K2 週前更新

Comps Analysis

anthropics

精選

以 Excel 試算表形式建立機構級可比公司分析，涵蓋營運指標、估值倍數與統計基準。

Data & Analytics39K2 週前更新

Audit Xls

anthropics

精選

稽核試算表公式與財務模型，找出錯誤、硬編碼和完整性問題並產出報告。

Data & Analytics39K2 週前更新