Nemo Curator
orchestra-research/ai-research-skills/05-data-processing/nemo-curatorby orchestra-research773a52944ba4MIT13K starsListed Oct 8, 2026Updated Oct 8, 2026Repository updated 3 months ago
GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.
Only the file list is public. File contents are available once the skill is installed in a workspace.
| Path | Size | Type |
|---|---|---|
| references/deduplication.md | 2.1 KB | text/markdown |
| references/filtering.md | 2.3 KB | text/markdown |
| SKILL.md | 9.1 KB | text/markdown |
Source and attribution
Source:orchestra-research/ai-research-skillsin05-data-processing/nemo-curatorat commit773a529
License: MIT
Content belongs to its original authors. SourceWeft indexes it from a public repository.
More from orchestra-research/ai-research-skills
Speculative Decoding
orchestra-research
Guides accelerating LLM inference with speculative decoding, Medusa multi-head prediction, and lookahead decoding.
Systems Paper Writing
orchestra-research
Guides paragraph-level structure and page budgets for writing systems conference papers such as OSDI, SOSP, ASPLOS and NSDI.
Moe Training
orchestra-research
Guides training and inference of Mixture of Experts models with DeepSpeed or HuggingFace, covering routing, load balancing and expert parallelism.
Presenting Conference Talks
orchestra-research
Generates conference presentation slides (Beamer LaTeX PDF and editable PPTX) from a compiled paper with speaker notes and talk script. Use when preparing oral talks, spotlight presentations, or invited talks for ML and systems conferences.
Model Pruning
orchestra-research
Guides LLM pruning with Wanda, SparseGPT, magnitude and N:M methods to cut model size and speed inference.
Outlines
orchestra-research
Guides use of the Outlines library for structured, schema-constrained text generation with local models.
More in Data & Analytics
Audit Xls
anthropics
Audits spreadsheet formulas and financial models for errors, hardcodes, and integrity issues, then reports findings.
Comps Analysis
anthropics
Builds institutional-grade comparable company analyses as Excel spreadsheets with operating metrics, valuation multiples and statistics.
Audit Xls
anthropics
Audits spreadsheet formulas and financial models for errors, hardcodes, and integrity issues, then reports findings.
Audit Xls
anthropics
Audits spreadsheet formulas and financial models for errors, hardcodes, and integrity issues, then reports findings.
Comps Analysis
anthropics
Builds institutional-grade comparable company analyses as Excel spreadsheets with operating metrics, valuation multiples and statistics.
Audit Xls
anthropics
Audits spreadsheet formulas and financial models for errors, hardcodes, and integrity issues, then reports findings.