Huggingface Tokenizers
orchestra-research/ai-research-skills/02-tokenization/huggingface-tokenizersby orchestra-research773a52944ba4MIT13K starsListed Oct 8, 2026Updated Oct 8, 2026Repository updated 3 months ago
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
Add to a SourceWeft workspace
- Open the skill in your dashboard and add it to a workspace.
- Enable it for the chats that should use it.
This skill is instructions only: it ships no scripts to execute.
Add to SourceWeftYou will be asked to sign in first, then taken straight to this skill.
Ask your agent to install it
Paste this prompt into Claude Code, Codex, Cursor or another agent that can run commands — or into SourceWeft chat. The agent reads this skill's install guide, shows you its source, license and scripts, and installs it with the SourceWeft CLI once you agree.
Read https://sourceweft.com/skills/gh-orchestra-research-ai-research-skills-huggingface-tokenizers/install.md and install the skill it describes. Before installing, show me its source, license and whether it ships scripts, and wait for my OK. Ask me before changing anything else on my machine.Install it yourself from a terminal
For Claude Code, Codex, Cursor and other local agents. The SourceWeft CLI fetches the skill from its source repository at the commit scanned here, and verifies every file against the hashes recorded when the skill was scanned. If anything differs, nothing is written.
npx @sourceweft/cli skills install @orchestra-research/huggingface-tokenizersAdd --agent claude-code, codex, cursor or universal to choose which agent gets it (Claude Code by default).
Upstream installer — not verified by SourceWeft
The open-source skills installer fetches the same pinned commit, but does not check the files against the hashes SourceWeft recorded.
npx skills add https://github.com/orchestra-research/ai-research-skills/tree/773a52944ba4747a18bd4ae9ade53fff041adcbc/02-tokenization/huggingface-tokenizersSource and attribution
Source:orchestra-research/ai-research-skillsin02-tokenization/huggingface-tokenizersat commit773a529
License: MIT
Content belongs to its original authors. SourceWeft indexes it from a public repository.
More from orchestra-research/ai-research-skills
Speculative Decoding
orchestra-research
Guides accelerating LLM inference with speculative decoding, Medusa multi-head prediction, and lookahead decoding.
Systems Paper Writing
orchestra-research
Guides paragraph-level structure and page budgets for writing systems conference papers such as OSDI, SOSP, ASPLOS and NSDI.
Moe Training
orchestra-research
Guides training and inference of Mixture of Experts models with DeepSpeed or HuggingFace, covering routing, load balancing and expert parallelism.
Presenting Conference Talks
orchestra-research
Generates conference presentation slides (Beamer LaTeX PDF and editable PPTX) from a compiled paper with speaker notes and talk script. Use when preparing oral talks, spotlight presentations, or invited talks for ML and systems conferences.
Model Pruning
orchestra-research
Guides LLM pruning with Wanda, SparseGPT, magnitude and N:M methods to cut model size and speed inference.
Outlines
orchestra-research
Guides use of the Outlines library for structured, schema-constrained text generation with local models.
More in Software Development
Playground
anthropics
Builds self-contained interactive HTML playgrounds with controls, live preview, and copyable prompt output.
M5 Onboard
anthropics
Provisions M5Stack ESP32 boards by detecting them on USB, flashing UIFlow 2.0 firmware, and installing a MicroPython app bundle.
Cardputer Buddy
anthropics
Guides iterating on a Cardputer-Adv MicroPython app bundle: adding apps, pushing files over USB-serial, tailing logs, and running REPL commands.
Web Design Guidelines
vercel-labs
Reviews UI code against Web Interface Guidelines fetched from a remote source and reports findings as file:line entries.
Vercel React Native Skills
vercel-labs
React Native and Expo best-practice rules covering list performance, animation, navigation, UI patterns, state, and monorepo setup.
Vercel React Best Practices
vercel-labs
Vercel's React and Next.js performance guidelines, organized as 70 rules for writing, reviewing, and refactoring code.