Huggingface Tokenizers

orchestra-research/ai-research-skills/02-tokenization/huggingface-tokenizers

作者 orchestra-research773a52944ba4MIT13K 个星标收录于 2026年10月8日更新于 2026年10月8日仓库3个月前更新

Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.

仅含说明Software Development
AI 生成的概览

指导使用 HuggingFace Tokenizers 进行快速分词、自定义分词器训练及 transformers 集成。

功能
该技能说明如何使用 HuggingFace Tokenizers 库进行快速、生产级的分词。内容涵盖加载预训练分词器、训练自定义 BPE、WordPiece 和 Unigram 词表、配置规范化、预分词和后处理流程，以及跟踪 token 对齐。它还展示了通过 AutoTokenizer 和 PreTrainedTokenizerFast 与 transformers 的集成。
适用场景
当你需要高性能分词、自定义分词器训练、对齐跟踪或生产级 NLP 流程时使用。它也适用于处理大型语料库或将分词器与 transformers 集成。
运行要求
需要 tokenizers、transformers 和 datasets Python 包。从 HuggingFace Hub 加载预训练分词器需要网络访问。该技能不附带脚本，仅为说明文档。

HuggingFace Tokenizers - Fast Tokenization for NLP

Fast, production-ready tokenizers with Rust performance and Python ease-of-use.

When to use HuggingFace Tokenizers

Use HuggingFace Tokenizers when:

  • Need extremely fast tokenization (<20s per GB of text)
  • Training custom tokenizers from scratch
  • Want alignment tracking (token → original text position)
  • Building production NLP pipelines
  • Need to tokenize large corpora efficiently

Performance:

  • Speed: <20 seconds to tokenize 1GB on CPU
  • Implementation: Rust core with Python/Node.js bindings
  • Efficiency: 10-100× faster than pure Python implementations

Use alternatives instead:

  • SentencePiece: Language-independent, used by T5/ALBERT
  • tiktoken: OpenAI's BPE tokenizer for GPT models
  • transformers AutoTokenizer: Loading pretrained only (uses this library internally)

Quick start

Installation

bash
# Install tokenizerspip install tokenizers
# With transformers integrationpip install tokenizers transformers

Load pretrained tokenizer

python
from tokenizers import Tokenizer
# Load from HuggingFace Hubtokenizer = Tokenizer.from_pretrained("bert-base-uncased")
# Encode textoutput = tokenizer.encode("Hello, how are you?")print(output.tokens)  # ['hello', ',', 'how', 'are', 'you', '?']print(output.ids)     # [7592, 1010, 2129, 2024, 2017, 1029]
# Decode backtext = tokenizer.decode(output.ids)print(text)  # "hello, how are you?"

Train custom BPE tokenizer

python
from tokenizers import Tokenizerfrom tokenizers.models import BPEfrom tokenizers.trainers import BpeTrainerfrom tokenizers.pre_tokenizers import Whitespace
# Initialize tokenizer with BPE modeltokenizer = Tokenizer(BPE(unk_token="[UNK]"))tokenizer.pre_tokenizer = Whitespace()
# Configure trainertrainer = BpeTrainer(    vocab_size=30000,    special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],    min_frequency=2)
# Train on filesfiles = ["train.txt", "validation.txt"]tokenizer.train(files, trainer)
# Savetokenizer.save("my-tokenizer.json")

Training time: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB

Batch encoding with padding

python
# Enable paddingtokenizer.enable_padding(pad_id=3, pad_token="[PAD]")
# Encode batchtexts = ["Hello world", "This is a longer sentence"]encodings = tokenizer.encode_batch(texts)
for encoding in encodings:    print(encoding.ids)# [101, 7592, 2088, 102, 3, 3, 3]# [101, 2023, 2003, 1037, 2936, 6251, 102]

Tokenization algorithms

BPE (Byte-Pair Encoding)

How it works:

  1. Start with character-level vocabulary
  2. Find most frequent character pair
  3. Merge into new token, add to vocabulary
  4. Repeat until vocabulary size reached

Used by: GPT-2, GPT-3, RoBERTa, BART, DeBERTa

python
from tokenizers import Tokenizerfrom tokenizers.models import BPEfrom tokenizers.trainers import BpeTrainerfrom tokenizers.pre_tokenizers import ByteLevel
tokenizer = Tokenizer(BPE(unk_token="<|endoftext|>"))tokenizer.pre_tokenizer = ByteLevel()
trainer = BpeTrainer(    vocab_size=50257,    special_tokens=["<|endoftext|>"],    min_frequency=2)
tokenizer.train(files=["data.txt"], trainer=trainer)

Advantages:

  • Handles OOV words well (breaks into subwords)
  • Flexible vocabulary size
  • Good for morphologically rich languages

Trade-offs:

  • Tokenization depends on merge order
  • May split common words unexpectedly

WordPiece

How it works:

  1. Start with character vocabulary
  2. Score merge pairs: frequency(pair) / (frequency(first) × frequency(second))
  3. Merge highest scoring pair
  4. Repeat until vocabulary size reached

Used by: BERT, DistilBERT, MobileBERT

python
from tokenizers import Tokenizerfrom tokenizers.models import WordPiecefrom tokenizers.trainers import WordPieceTrainerfrom tokenizers.pre_tokenizers import Whitespacefrom tokenizers.normalizers import BertNormalizer
tokenizer = Tokenizer(WordPiece(unk_token="[UNK]"))tokenizer.normalizer = BertNormalizer(lowercase=True)tokenizer.pre_tokenizer = Whitespace()
trainer = WordPieceTrainer(    vocab_size=30522,    special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],    continuing_subword_prefix="##")
tokenizer.train(files=["corpus.txt"], trainer=trainer)

Advantages:

  • Prioritizes meaningful merges (high score = semantically related)
  • Used successfully in BERT (state-of-the-art results)

Trade-offs:

  • Unknown words become [UNK] if no subword match
  • Saves vocabulary, not merge rules (larger files)

Unigram

How it works:

  1. Start with large vocabulary (all substrings)
  2. Compute loss for corpus with current vocabulary
  3. Remove tokens with minimal impact on loss
  4. Repeat until vocabulary size reached

Used by: ALBERT, T5, mBART, XLNet (via SentencePiece)

python
from tokenizers import Tokenizerfrom tokenizers.models import Unigramfrom tokenizers.trainers import UnigramTrainer
tokenizer = Tokenizer(Unigram())
trainer = UnigramTrainer(    vocab_size=8000,    special_tokens=["<unk>", "<s>", "</s>"],    unk_token="<unk>")
tokenizer.train(files=["data.txt"], trainer=trainer)

Advantages:

  • Probabilistic (finds most likely tokenization)
  • Works well for languages without word boundaries
  • Handles diverse linguistic contexts

Trade-offs:

  • Computationally expensive to train
  • More hyperparameters to tune

Tokenization pipeline

Complete pipeline: Normalization → Pre-tokenization → Model → Post-processing

Normalization

Clean and standardize text:

python
from tokenizers.normalizers import NFD, StripAccents, Lowercase, Sequence
tokenizer.normalizer = Sequence([    NFD(),           # Unicode normalization (decompose)    Lowercase(),     # Convert to lowercase    StripAccents()   # Remove accents])
# Input: "Héllo WORLD"# After normalization: "hello world"

Common normalizers:

  • NFD, NFC, NFKD, NFKC - Unicode normalization forms
  • Lowercase() - Convert to lowercase
  • StripAccents() - Remove accents (é → e)
  • Strip() - Remove whitespace
  • Replace(pattern, content) - Regex replacement

Pre-tokenization

Split text into word-like units:

python
from tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence, ByteLevel
# Split on whitespace and punctuationtokenizer.pre_tokenizer = Sequence([    Whitespace(),    Punctuation()])
# Input: "Hello, world!"# After pre-tokenization: ["Hello", ",", "world", "!"]

Common pre-tokenizers:

  • Whitespace() - Split on spaces, tabs, newlines
  • ByteLevel() - GPT-2 style byte-level splitting
  • Punctuation() - Isolate punctuation
  • Digits(individual_digits=True) - Split digits individually
  • Metaspace() - Replace spaces with ▁ (SentencePiece style)

Post-processing

Add special tokens for model input:

python
from tokenizers.processors import TemplateProcessing
# BERT-style: [CLS] sentence [SEP]tokenizer.post_processor = TemplateProcessing(    single="[CLS] $A [SEP]",    pair="[CLS] $A [SEP] $B [SEP]",    special_tokens=[        ("[CLS]", 1),        ("[SEP]", 2),    ],)

Common patterns:

python
# GPT-2: sentence <|endoftext|>TemplateProcessing(    single="$A <|endoftext|>",    special_tokens=[("<|endoftext|>", 50256)])
# RoBERTa: <s> sentence </s>TemplateProcessing(    single="<s> $A </s>",    pair="<s> $A </s> </s> $B </s>",    special_tokens=[("<s>", 0), ("</s>", 2)])

Alignment tracking

Track token positions in original text:

python
output = tokenizer.encode("Hello, world!")
# Get token offsetsfor token, offset in zip(output.tokens, output.offsets):    start, end = offset    print(f"{token:10} → [{start:2}, {end:2}): {text[start:end]!r}")
# Output:# hello      → [ 0,  5): 'Hello'# ,          → [ 5,  6): ','# world      → [ 7, 12): 'world'# !          → [12, 13): '!'

Use cases:

  • Named entity recognition (map predictions back to text)
  • Question answering (extract answer spans)
  • Token classification (align labels to original positions)

Integration with transformers

Load with AutoTokenizer

python
from transformers import AutoTokenizer
# AutoTokenizer automatically uses fast tokenizerstokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
# Check if using fast tokenizerprint(tokenizer.is_fast)  # True
# Access underlying tokenizers.Tokenizerfast_tokenizer = tokenizer.backend_tokenizerprint(type(fast_tokenizer))  # <class 'tokenizers.Tokenizer'>

Convert custom tokenizer to transformers

python
from tokenizers import Tokenizerfrom transformers import PreTrainedTokenizerFast
# Train custom tokenizertokenizer = Tokenizer(BPE())# ... train tokenizer ...tokenizer.save("my-tokenizer.json")
# Wrap for transformerstransformers_tokenizer = PreTrainedTokenizerFast(    tokenizer_file="my-tokenizer.json",    unk_token="[UNK]",    pad_token="[PAD]",    cls_token="[CLS]",    sep_token="[SEP]",    mask_token="[MASK]")
# Use like any transformers tokenizeroutputs = transformers_tokenizer(    "Hello world",    padding=True,    truncation=True,    max_length=512,    return_tensors="pt")

Common patterns

Train from iterator (large datasets)

python
from datasets import load_dataset
# Load datasetdataset = load_dataset("wikitext", "wikitext-103-raw-v1", split="train")
# Create batch iteratordef batch_iterator(batch_size=1000):    for i in range(0, len(dataset), batch_size):        yield dataset[i:i + batch_size]["text"]
# Train tokenizertokenizer.train_from_iterator(    batch_iterator(),    trainer=trainer,    length=len(dataset)  # For progress bar)

Performance: Processes 1GB in ~10-20 minutes

Enable truncation and padding

python
# Enable truncationtokenizer.enable_truncation(max_length=512)
# Enable paddingtokenizer.enable_padding(    pad_id=tokenizer.token_to_id("[PAD]"),    pad_token="[PAD]",    length=512  # Fixed length, or None for batch max)
# Encode with bothoutput = tokenizer.encode("This is a long sentence that will be truncated...")print(len(output.ids))  # 512

Multi-processing

python
from tokenizers import Tokenizerfrom multiprocessing import Pool
# Load tokenizertokenizer = Tokenizer.from_file("tokenizer.json")
def encode_batch(texts):    return tokenizer.encode_batch(texts)
# Process large corpus in parallelwith Pool(8) as pool:    # Split corpus into chunks    chunk_size = 1000    chunks = [corpus[i:i+chunk_size] for i in range(0, len(corpus), chunk_size)]
    # Encode in parallel    results = pool.map(encode_batch, chunks)

Speedup: 5-8× with 8 cores

Performance benchmarks

Training speed

Corpus SizeBPE (30k vocab)WordPiece (30k)Unigram (8k)
10 MB15 sec18 sec25 sec
100 MB1.5 min2 min4 min
1 GB15 min20 min40 min

Hardware: 16-core CPU, tested on English Wikipedia

Tokenization speed

Implementation1 GB corpusThroughput
Pure Python~20 minutes~50 MB/min
HF Tokenizers~15 seconds~4 GB/min
Speedup80×80×

Test: English text, average sentence length 20 words

Memory usage

TaskMemory
Load tokenizer~10 MB
Train BPE (30k vocab)~200 MB
Encode 1M sentences~500 MB

Supported models

Pre-trained tokenizers available via from_pretrained():

BERT family:

  • bert-base-uncased, bert-large-cased
  • distilbert-base-uncased
  • roberta-base, roberta-large

GPT family:

  • gpt2, gpt2-medium, gpt2-large
  • distilgpt2

T5 family:

  • t5-small, t5-base, t5-large
  • google/flan-t5-xxl

Other:

  • facebook/bart-base, facebook/mbart-large-cc25
  • albert-base-v2, albert-xlarge-v2
  • xlm-roberta-base, xlm-roberta-large

Browse all: https://huggingface.co/models?library=tokenizers

References

  • Training Guide [blocked] - Train custom tokenizers, configure trainers, handle large datasets
  • Algorithms Deep Dive [blocked] - BPE, WordPiece, Unigram explained in detail
  • Pipeline Components [blocked] - Normalizers, pre-tokenizers, post-processors, decoders
  • Transformers Integration [blocked] - AutoTokenizer, PreTrainedTokenizerFast, special tokens

Resources

来源与署名

来源：orchestra-research/ai-research-skills位于02-tokenization/huggingface-tokenizers提交773a529

许可证： MIT

内容归原作者所有。SourceWeft 从公开仓库中收录这些内容。

举报或申请下架

更多来自 orchestra-research/ai-research-skills 的技能

Speculative Decoding

orchestra-research

指导使用推测解码、Medusa 多头预测和前瞻解码来加速大语言模型推理。

AI & Agents13K3个月前更新

Systems Paper Writing

orchestra-research

为 OSDI、SOSP、ASPLOS、NSDI 等系统会议论文写作提供段落级结构与篇幅分配指导。

Writing & Content13K3个月前更新

Moe Training

orchestra-research

指导使用 DeepSpeed 或 HuggingFace 训练与推理混合专家模型，涵盖路由、负载均衡与专家并行。

AI & Agents13K3个月前更新

Presenting Conference Talks

orchestra-research

Generates conference presentation slides (Beamer LaTeX PDF and editable PPTX) from a compiled paper with speaker notes and talk script. Use when preparing oral talks, spotlight presentations, or invited talks for ML and systems conferences.

待分类13K3个月前更新

Model Pruning

orchestra-research

指导使用 Wanda、SparseGPT、幅度剪枝和 N:M 方法压缩 LLM，减小模型体积并加速推理。

AI & Agents13K3个月前更新

Outlines

orchestra-research

指导使用 Outlines 库，通过本地模型进行受结构约束的文本生成。

AI & Agents13K3个月前更新

更多Software Development技能

Playground

anthropics

精选

构建自包含的交互式 HTML 演练场，包含控件、实时预览和可复制的提示词输出。

Software Development37K今天更新

M5 Onboard

anthropics

精选

通过 USB 检测 M5Stack ESP32 开发板，刷写 UIFlow 2.0 固件并安装 MicroPython 应用包。

Software Development37K今天更新

Cardputer Buddy

anthropics

精选

指导迭代 Cardputer-Adv 的 MicroPython 应用包：添加应用、通过 USB 串口推送文件、查看日志并运行 REPL 命令。

Software Development37K今天更新

Web Design Guidelines

vercel-labs

精选

依据从远程来源获取的 Web Interface Guidelines 审查 UI 代码，并以 file:line 形式报告问题。

Software Development32K5周前更新

Vercel React Native Skills

vercel-labs

精选

React Native 与 Expo 最佳实践规则，涵盖列表性能、动画、导航、UI 模式、状态与 monorepo 配置。

Software Development32K5周前更新

Vercel React Best Practices

vercel-labs

精选

Vercel 的 React 与 Next.js 性能指南，以 70 条规则的形式用于编写、审查和重构代码。

Software Development32K5周前更新