Huggingface Tokenizers
orchestra-research/ai-research-skills/02-tokenization/huggingface-tokenizersby orchestra-research773a52944ba4MIT13K starsListed Oct 8, 2026Updated Oct 8, 2026Repository updated 3 months ago
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
Only the file list is public. File contents are available once the skill is installed in a workspace.
| Path | Size | Type |
|---|---|---|
| references/algorithms.md | 14.8 KB | text/markdown |
| references/integration.md | 15 KB | text/markdown |
| references/pipeline.md | 16.4 KB | text/markdown |
| references/training.md | 14.2 KB | text/markdown |
| SKILL.md | 13.4 KB | text/markdown |
Source and attribution
Source:orchestra-research/ai-research-skillsin02-tokenization/huggingface-tokenizersat commit773a529
License: MIT
Content belongs to its original authors. SourceWeft indexes it from a public repository.
More from orchestra-research/ai-research-skills
Speculative Decoding
orchestra-research
Guides accelerating LLM inference with speculative decoding, Medusa multi-head prediction, and lookahead decoding.
Systems Paper Writing
orchestra-research
Guides paragraph-level structure and page budgets for writing systems conference papers such as OSDI, SOSP, ASPLOS and NSDI.
Moe Training
orchestra-research
Guides training and inference of Mixture of Experts models with DeepSpeed or HuggingFace, covering routing, load balancing and expert parallelism.
Presenting Conference Talks
orchestra-research
Generates conference presentation slides (Beamer LaTeX PDF and editable PPTX) from a compiled paper with speaker notes and talk script. Use when preparing oral talks, spotlight presentations, or invited talks for ML and systems conferences.
Model Pruning
orchestra-research
Guides LLM pruning with Wanda, SparseGPT, magnitude and N:M methods to cut model size and speed inference.
Outlines
orchestra-research
Guides use of the Outlines library for structured, schema-constrained text generation with local models.
More in Software Development
Playground
anthropics
Builds self-contained interactive HTML playgrounds with controls, live preview, and copyable prompt output.
M5 Onboard
anthropics
Provisions M5Stack ESP32 boards by detecting them on USB, flashing UIFlow 2.0 firmware, and installing a MicroPython app bundle.
Cardputer Buddy
anthropics
Guides iterating on a Cardputer-Adv MicroPython app bundle: adding apps, pushing files over USB-serial, tailing logs, and running REPL commands.
Web Design Guidelines
vercel-labs
Reviews UI code against Web Interface Guidelines fetched from a remote source and reports findings as file:line entries.
Vercel React Native Skills
vercel-labs
React Native and Expo best-practice rules covering list performance, animation, navigation, UI patterns, state, and monorepo setup.
Vercel React Best Practices
vercel-labs
Vercel's React and Next.js performance guidelines, organized as 70 rules for writing, reviewing, and refactoring code.