Huggingface Tokenizers

orchestra-research/ai-research-skills/02-tokenization/huggingface-tokenizers

by orchestra-research773a52944ba4MIT13K starsListed Oct 8, 2026Updated Oct 8, 2026Repository updated 3 months ago

Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.

Instructions onlySoftware Development

AI-generated overview

Guides use of HuggingFace Tokenizers for fast tokenization, custom tokenizer training, and transformers integration.

What it does: This skill explains how to use the HuggingFace Tokenizers library for fast, production-ready tokenization. It covers loading pretrained tokenizers, training custom BPE, WordPiece, and Unigram vocabularies, configuring normalization, pre-tokenization, and post-processing pipelines, and tracking token alignments. It also shows integration with transformers via AutoTokenizer and PreTrainedTokenizerFast.
When to use it: Use it when you need high-performance tokenization, custom tokenizer training, alignment tracking, or production NLP pipelines. It is also relevant when working with large corpora or integrating tokenizers with transformers.
Requirements: Requires the tokenizers, transformers, and datasets Python packages. Network access is needed to load pretrained tokenizers from the HuggingFace Hub. The skill ships no scripts; it is instructions only.

HuggingFace Tokenizers - Fast Tokenization for NLP

Fast, production-ready tokenizers with Rust performance and Python ease-of-use.

When to use HuggingFace Tokenizers

Use HuggingFace Tokenizers when:

Need extremely fast tokenization (<20s per GB of text)
Training custom tokenizers from scratch
Want alignment tracking (token → original text position)
Building production NLP pipelines
Need to tokenize large corpora efficiently

Performance:

Speed: <20 seconds to tokenize 1GB on CPU
Implementation: Rust core with Python/Node.js bindings
Efficiency: 10-100× faster than pure Python implementations

Use alternatives instead:

SentencePiece: Language-independent, used by T5/ALBERT
tiktoken: OpenAI's BPE tokenizer for GPT models
transformers AutoTokenizer: Loading pretrained only (uses this library internally)

Quick start

Installation

bash

# Install tokenizerspip install tokenizers
# With transformers integrationpip install tokenizers transformers

# Install tokenizerspip install tokenizers
# With transformers integrationpip install tokenizers transformers

Load pretrained tokenizer

python

from tokenizers import Tokenizer
# Load from HuggingFace Hubtokenizer = Tokenizer.from_pretrained("bert-base-uncased")
# Encode textoutput = tokenizer.encode("Hello, how are you?")print(output.tokens)  # ['hello', ',', 'how', 'are', 'you', '?']print(output.ids)     # [7592, 1010, 2129, 2024, 2017, 1029]
# Decode backtext = tokenizer.decode(output.ids)print(text)  # "hello, how are you?"

from tokenizers import Tokenizer
# Load from HuggingFace Hubtokenizer = Tokenizer.from_pretrained("bert-base-uncased")
# Encode textoutput = tokenizer.encode("Hello, how are you?")print(output.tokens)  # ['hello', ',', 'how', 'are', 'you', '?']print(output.ids)     # [7592, 1010, 2129, 2024, 2017, 1029]
# Decode backtext = tokenizer.decode(output.ids)print(text)  # "hello, how are you?"

Train custom BPE tokenizer

python

from tokenizers import Tokenizerfrom tokenizers.models import BPEfrom tokenizers.trainers import BpeTrainerfrom tokenizers.pre_tokenizers import Whitespace
# Initialize tokenizer with BPE modeltokenizer = Tokenizer(BPE(unk_token="[UNK]"))tokenizer.pre_tokenizer = Whitespace()
# Configure trainertrainer = BpeTrainer(    vocab_size=30000,    special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],    min_frequency=2)
# Train on filesfiles = ["train.txt", "validation.txt"]tokenizer.train(files, trainer)
# Savetokenizer.save("my-tokenizer.json")

from tokenizers import Tokenizerfrom tokenizers.models import BPEfrom tokenizers.trainers import BpeTrainerfrom tokenizers.pre_tokenizers import Whitespace
# Initialize tokenizer with BPE modeltokenizer = Tokenizer(BPE(unk_token="[UNK]"))tokenizer.pre_tokenizer = Whitespace()
# Configure trainertrainer = BpeTrainer(    vocab_size=30000,    special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],    min_frequency=2)
# Train on filesfiles = ["train.txt", "validation.txt"]tokenizer.train(files, trainer)
# Savetokenizer.save("my-tokenizer.json")

Training time: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB

Batch encoding with padding

python

# Enable paddingtokenizer.enable_padding(pad_id=3, pad_token="[PAD]")
# Encode batchtexts = ["Hello world", "This is a longer sentence"]encodings = tokenizer.encode_batch(texts)
for encoding in encodings:    print(encoding.ids)# [101, 7592, 2088, 102, 3, 3, 3]# [101, 2023, 2003, 1037, 2936, 6251, 102]

# Enable paddingtokenizer.enable_padding(pad_id=3, pad_token="[PAD]")
# Encode batchtexts = ["Hello world", "This is a longer sentence"]encodings = tokenizer.encode_batch(texts)
for encoding in encodings:    print(encoding.ids)# [101, 7592, 2088, 102, 3, 3, 3]# [101, 2023, 2003, 1037, 2936, 6251, 102]

Tokenization algorithms

BPE (Byte-Pair Encoding)

How it works:

Start with character-level vocabulary
Find most frequent character pair
Merge into new token, add to vocabulary
Repeat until vocabulary size reached

Used by: GPT-2, GPT-3, RoBERTa, BART, DeBERTa

python

from tokenizers import Tokenizerfrom tokenizers.models import BPEfrom tokenizers.trainers import BpeTrainerfrom tokenizers.pre_tokenizers import ByteLevel
tokenizer = Tokenizer(BPE(unk_token="<|endoftext|>"))tokenizer.pre_tokenizer = ByteLevel()
trainer = BpeTrainer(    vocab_size=50257,    special_tokens=["<|endoftext|>"],    min_frequency=2)
tokenizer.train(files=["data.txt"], trainer=trainer)

from tokenizers import Tokenizerfrom tokenizers.models import BPEfrom tokenizers.trainers import BpeTrainerfrom tokenizers.pre_tokenizers import ByteLevel
tokenizer = Tokenizer(BPE(unk_token="<|endoftext|>"))tokenizer.pre_tokenizer = ByteLevel()
trainer = BpeTrainer(    vocab_size=50257,    special_tokens=["<|endoftext|>"],    min_frequency=2)
tokenizer.train(files=["data.txt"], trainer=trainer)

Advantages:

Handles OOV words well (breaks into subwords)
Flexible vocabulary size
Good for morphologically rich languages

Trade-offs:

Tokenization depends on merge order
May split common words unexpectedly

WordPiece

How it works:

Start with character vocabulary
Score merge pairs: frequency(pair) / (frequency(first) × frequency(second))
Merge highest scoring pair
Repeat until vocabulary size reached

Used by: BERT, DistilBERT, MobileBERT

python

from tokenizers import Tokenizerfrom tokenizers.models import WordPiecefrom tokenizers.trainers import WordPieceTrainerfrom tokenizers.pre_tokenizers import Whitespacefrom tokenizers.normalizers import BertNormalizer
tokenizer = Tokenizer(WordPiece(unk_token="[UNK]"))tokenizer.normalizer = BertNormalizer(lowercase=True)tokenizer.pre_tokenizer = Whitespace()
trainer = WordPieceTrainer(    vocab_size=30522,    special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],    continuing_subword_prefix="##")
tokenizer.train(files=["corpus.txt"], trainer=trainer)

from tokenizers import Tokenizerfrom tokenizers.models import WordPiecefrom tokenizers.trainers import WordPieceTrainerfrom tokenizers.pre_tokenizers import Whitespacefrom tokenizers.normalizers import BertNormalizer
tokenizer = Tokenizer(WordPiece(unk_token="[UNK]"))tokenizer.normalizer = BertNormalizer(lowercase=True)tokenizer.pre_tokenizer = Whitespace()
trainer = WordPieceTrainer(    vocab_size=30522,    special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],    continuing_subword_prefix="##")
tokenizer.train(files=["corpus.txt"], trainer=trainer)

Advantages:

Prioritizes meaningful merges (high score = semantically related)
Used successfully in BERT (state-of-the-art results)

Trade-offs:

Unknown words become [UNK] if no subword match
Saves vocabulary, not merge rules (larger files)

Unigram

How it works:

Start with large vocabulary (all substrings)
Compute loss for corpus with current vocabulary
Remove tokens with minimal impact on loss
Repeat until vocabulary size reached

Used by: ALBERT, T5, mBART, XLNet (via SentencePiece)

python

from tokenizers import Tokenizerfrom tokenizers.models import Unigramfrom tokenizers.trainers import UnigramTrainer
tokenizer = Tokenizer(Unigram())
trainer = UnigramTrainer(    vocab_size=8000,    special_tokens=["<unk>", "<s>", "</s>"],    unk_token="<unk>")
tokenizer.train(files=["data.txt"], trainer=trainer)

from tokenizers import Tokenizerfrom tokenizers.models import Unigramfrom tokenizers.trainers import UnigramTrainer
tokenizer = Tokenizer(Unigram())
trainer = UnigramTrainer(    vocab_size=8000,    special_tokens=["<unk>", "<s>", "</s>"],    unk_token="<unk>")
tokenizer.train(files=["data.txt"], trainer=trainer)

Advantages:

Probabilistic (finds most likely tokenization)
Works well for languages without word boundaries
Handles diverse linguistic contexts

Trade-offs:

Computationally expensive to train
More hyperparameters to tune

Tokenization pipeline

Complete pipeline: Normalization → Pre-tokenization → Model → Post-processing

Normalization

Clean and standardize text:

python

from tokenizers.normalizers import NFD, StripAccents, Lowercase, Sequence
tokenizer.normalizer = Sequence([    NFD(),           # Unicode normalization (decompose)    Lowercase(),     # Convert to lowercase    StripAccents()   # Remove accents])
# Input: "Héllo WORLD"# After normalization: "hello world"

from tokenizers.normalizers import NFD, StripAccents, Lowercase, Sequence
tokenizer.normalizer = Sequence([    NFD(),           # Unicode normalization (decompose)    Lowercase(),     # Convert to lowercase    StripAccents()   # Remove accents])
# Input: "Héllo WORLD"# After normalization: "hello world"

Common normalizers:

NFD, NFC, NFKD, NFKC - Unicode normalization forms
Lowercase() - Convert to lowercase
StripAccents() - Remove accents (é → e)
Strip() - Remove whitespace
Replace(pattern, content) - Regex replacement

Pre-tokenization

Split text into word-like units:

python

from tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence, ByteLevel
# Split on whitespace and punctuationtokenizer.pre_tokenizer = Sequence([    Whitespace(),    Punctuation()])
# Input: "Hello, world!"# After pre-tokenization: ["Hello", ",", "world", "!"]

from tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence, ByteLevel
# Split on whitespace and punctuationtokenizer.pre_tokenizer = Sequence([    Whitespace(),    Punctuation()])
# Input: "Hello, world!"# After pre-tokenization: ["Hello", ",", "world", "!"]

Common pre-tokenizers:

Whitespace() - Split on spaces, tabs, newlines
ByteLevel() - GPT-2 style byte-level splitting
Punctuation() - Isolate punctuation
Digits(individual_digits=True) - Split digits individually
Metaspace() - Replace spaces with ▁ (SentencePiece style)

Post-processing

Add special tokens for model input:

python

from tokenizers.processors import TemplateProcessing
# BERT-style: [CLS] sentence [SEP]tokenizer.post_processor = TemplateProcessing(    single="[CLS] $A [SEP]",    pair="[CLS] $A [SEP] $B [SEP]",    special_tokens=[        ("[CLS]", 1),        ("[SEP]", 2),    ],)

from tokenizers.processors import TemplateProcessing
# BERT-style: [CLS] sentence [SEP]tokenizer.post_processor = TemplateProcessing(    single="[CLS] $A [SEP]",    pair="[CLS] $A [SEP] $B [SEP]",    special_tokens=[        ("[CLS]", 1),        ("[SEP]", 2),    ],)

Common patterns:

python

# GPT-2: sentence <|endoftext|>TemplateProcessing(    single="$A <|endoftext|>",    special_tokens=[("<|endoftext|>", 50256)])
# RoBERTa: <s> sentence </s>TemplateProcessing(    single="<s> $A </s>",    pair="<s> $A </s> </s> $B </s>",    special_tokens=[("<s>", 0), ("</s>", 2)])

# GPT-2: sentence <|endoftext|>TemplateProcessing(    single="$A <|endoftext|>",    special_tokens=[("<|endoftext|>", 50256)])
# RoBERTa: <s> sentence </s>TemplateProcessing(    single="<s> $A </s>",    pair="<s> $A </s> </s> $B </s>",    special_tokens=[("<s>", 0), ("</s>", 2)])

Alignment tracking

Track token positions in original text:

python

output = tokenizer.encode("Hello, world!")
# Get token offsetsfor token, offset in zip(output.tokens, output.offsets):    start, end = offset    print(f"{token:10} → [{start:2}, {end:2}): {text[start:end]!r}")
# Output:# hello      → [ 0,  5): 'Hello'# ,          → [ 5,  6): ','# world      → [ 7, 12): 'world'# !          → [12, 13): '!'

output = tokenizer.encode("Hello, world!")
# Get token offsetsfor token, offset in zip(output.tokens, output.offsets):    start, end = offset    print(f"{token:10} → [{start:2}, {end:2}): {text[start:end]!r}")
# Output:# hello      → [ 0,  5): 'Hello'# ,          → [ 5,  6): ','# world      → [ 7, 12): 'world'# !          → [12, 13): '!'

Use cases:

Named entity recognition (map predictions back to text)
Question answering (extract answer spans)
Token classification (align labels to original positions)

Integration with transformers

Load with AutoTokenizer

python

from transformers import AutoTokenizer
# AutoTokenizer automatically uses fast tokenizerstokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
# Check if using fast tokenizerprint(tokenizer.is_fast)  # True
# Access underlying tokenizers.Tokenizerfast_tokenizer = tokenizer.backend_tokenizerprint(type(fast_tokenizer))  # <class 'tokenizers.Tokenizer'>

from transformers import AutoTokenizer
# AutoTokenizer automatically uses fast tokenizerstokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
# Check if using fast tokenizerprint(tokenizer.is_fast)  # True
# Access underlying tokenizers.Tokenizerfast_tokenizer = tokenizer.backend_tokenizerprint(type(fast_tokenizer))  # <class 'tokenizers.Tokenizer'>

Convert custom tokenizer to transformers

python

from tokenizers import Tokenizerfrom transformers import PreTrainedTokenizerFast
# Train custom tokenizertokenizer = Tokenizer(BPE())# ... train tokenizer ...tokenizer.save("my-tokenizer.json")
# Wrap for transformerstransformers_tokenizer = PreTrainedTokenizerFast(    tokenizer_file="my-tokenizer.json",    unk_token="[UNK]",    pad_token="[PAD]",    cls_token="[CLS]",    sep_token="[SEP]",    mask_token="[MASK]")
# Use like any transformers tokenizeroutputs = transformers_tokenizer(    "Hello world",    padding=True,    truncation=True,    max_length=512,    return_tensors="pt")

from tokenizers import Tokenizerfrom transformers import PreTrainedTokenizerFast
# Train custom tokenizertokenizer = Tokenizer(BPE())# ... train tokenizer ...tokenizer.save("my-tokenizer.json")
# Wrap for transformerstransformers_tokenizer = PreTrainedTokenizerFast(    tokenizer_file="my-tokenizer.json",    unk_token="[UNK]",    pad_token="[PAD]",    cls_token="[CLS]",    sep_token="[SEP]",    mask_token="[MASK]")
# Use like any transformers tokenizeroutputs = transformers_tokenizer(    "Hello world",    padding=True,    truncation=True,    max_length=512,    return_tensors="pt")

Common patterns

Train from iterator (large datasets)

python

from datasets import load_dataset
# Load datasetdataset = load_dataset("wikitext", "wikitext-103-raw-v1", split="train")
# Create batch iteratordef batch_iterator(batch_size=1000):    for i in range(0, len(dataset), batch_size):        yield dataset[i:i + batch_size]["text"]
# Train tokenizertokenizer.train_from_iterator(    batch_iterator(),    trainer=trainer,    length=len(dataset)  # For progress bar)

from datasets import load_dataset
# Load datasetdataset = load_dataset("wikitext", "wikitext-103-raw-v1", split="train")
# Create batch iteratordef batch_iterator(batch_size=1000):    for i in range(0, len(dataset), batch_size):        yield dataset[i:i + batch_size]["text"]
# Train tokenizertokenizer.train_from_iterator(    batch_iterator(),    trainer=trainer,    length=len(dataset)  # For progress bar)

Performance: Processes 1GB in ~10-20 minutes

Enable truncation and padding

python

# Enable truncationtokenizer.enable_truncation(max_length=512)
# Enable paddingtokenizer.enable_padding(    pad_id=tokenizer.token_to_id("[PAD]"),    pad_token="[PAD]",    length=512  # Fixed length, or None for batch max)
# Encode with bothoutput = tokenizer.encode("This is a long sentence that will be truncated...")print(len(output.ids))  # 512

# Enable truncationtokenizer.enable_truncation(max_length=512)
# Enable paddingtokenizer.enable_padding(    pad_id=tokenizer.token_to_id("[PAD]"),    pad_token="[PAD]",    length=512  # Fixed length, or None for batch max)
# Encode with bothoutput = tokenizer.encode("This is a long sentence that will be truncated...")print(len(output.ids))  # 512

Multi-processing

python

from tokenizers import Tokenizerfrom multiprocessing import Pool
# Load tokenizertokenizer = Tokenizer.from_file("tokenizer.json")
def encode_batch(texts):    return tokenizer.encode_batch(texts)
# Process large corpus in parallelwith Pool(8) as pool:    # Split corpus into chunks    chunk_size = 1000    chunks = [corpus[i:i+chunk_size] for i in range(0, len(corpus), chunk_size)]
    # Encode in parallel    results = pool.map(encode_batch, chunks)

from tokenizers import Tokenizerfrom multiprocessing import Pool
# Load tokenizertokenizer = Tokenizer.from_file("tokenizer.json")
def encode_batch(texts):    return tokenizer.encode_batch(texts)
# Process large corpus in parallelwith Pool(8) as pool:    # Split corpus into chunks    chunk_size = 1000    chunks = [corpus[i:i+chunk_size] for i in range(0, len(corpus), chunk_size)]
    # Encode in parallel    results = pool.map(encode_batch, chunks)

Speedup: 5-8× with 8 cores

Performance benchmarks

Training speed

Hardware: 16-core CPU, tested on English Wikipedia

Tokenization speed

Test: English text, average sentence length 20 words

Memory usage

Supported models

Pre-trained tokenizers available via from_pretrained():

BERT family:

bert-base-uncased, bert-large-cased
distilbert-base-uncased
roberta-base, roberta-large

GPT family:

gpt2, gpt2-medium, gpt2-large
distilgpt2

T5 family:

t5-small, t5-base, t5-large
google/flan-t5-xxl

Other:

facebook/bart-base, facebook/mbart-large-cc25
albert-base-v2, albert-xlarge-v2
xlm-roberta-base, xlm-roberta-large

Browse all: https://huggingface.co/models?library=tokenizers

References

Training Guide [blocked] - Train custom tokenizers, configure trainers, handle large datasets
Algorithms Deep Dive [blocked] - BPE, WordPiece, Unigram explained in detail
Pipeline Components [blocked] - Normalizers, pre-tokenizers, post-processors, decoders
Transformers Integration [blocked] - AutoTokenizer, PreTrainedTokenizerFast, special tokens

Resources

Docs: https://huggingface.co/docs/tokenizers
GitHub: https://github.com/huggingface/tokenizers ⭐ 9,000+
Version: 0.20.0+
Course: https://huggingface.co/learn/nlp-course/chapter6/1
Paper: BPE (Sennrich et al., 2016), WordPiece (Schuster & Nakajima, 2012)

Source and attribution

Source:orchestra-research/ai-research-skillsin02-tokenization/huggingface-tokenizersat commit773a529

License: MIT

Content belongs to its original authors. SourceWeft indexes it from a public repository.

Report or request removal

More from orchestra-research/ai-research-skills

Speculative Decoding

orchestra-research

Guides accelerating LLM inference with speculative decoding, Medusa multi-head prediction, and lookahead decoding.

AI & Agents13Kupdated 3 months ago

Systems Paper Writing

orchestra-research

Guides paragraph-level structure and page budgets for writing systems conference papers such as OSDI, SOSP, ASPLOS and NSDI.

Writing & Content13Kupdated 3 months ago

Moe Training

orchestra-research

Guides training and inference of Mixture of Experts models with DeepSpeed or HuggingFace, covering routing, load balancing and expert parallelism.

AI & Agents13Kupdated 3 months ago

Presenting Conference Talks

orchestra-research

Generates conference presentation slides (Beamer LaTeX PDF and editable PPTX) from a compiled paper with speaker notes and talk script. Use when preparing oral talks, spotlight presentations, or invited talks for ML and systems conferences.

Awaiting classification13Kupdated 3 months ago

Model Pruning

orchestra-research

Guides LLM pruning with Wanda, SparseGPT, magnitude and N:M methods to cut model size and speed inference.

AI & Agents13Kupdated 3 months ago

Outlines

orchestra-research

Guides use of the Outlines library for structured, schema-constrained text generation with local models.

AI & Agents13Kupdated 3 months ago

Corpus Size	BPE (30k vocab)	WordPiece (30k)	Unigram (8k)
10 MB	15 sec	18 sec	25 sec
100 MB	1.5 min	2 min	4 min
1 GB	15 min	20 min	40 min

Implementation	1 GB corpus	Throughput
Pure Python	~20 minutes	~50 MB/min
HF Tokenizers	~15 seconds	~4 GB/min
Speedup	80×	80×

Task	Memory
Load tokenizer	~10 MB
Train BPE (30k vocab)	~200 MB
Encode 1M sentences	~500 MB

Huggingface Tokenizers

HuggingFace Tokenizers - Fast Tokenization for NLP

When to use HuggingFace Tokenizers

Quick start

Installation

Load pretrained tokenizer

Train custom BPE tokenizer

Batch encoding with padding

Tokenization algorithms

BPE (Byte-Pair Encoding)

WordPiece

Unigram

Tokenization pipeline

Normalization

Pre-tokenization

Post-processing

Alignment tracking

Integration with transformers

Load with AutoTokenizer

Convert custom tokenizer to transformers

Common patterns

Train from iterator (large datasets)

Enable truncation and padding

Multi-processing

Performance benchmarks

Training speed

Tokenization speed

Memory usage

Supported models

References

Resources

Source and attribution

More from orchestra-research/ai-research-skills

Speculative Decoding

Systems Paper Writing

Moe Training

Presenting Conference Talks

Model Pruning

Outlines

More in Software Development

Playground

M5 Onboard

Cardputer Buddy

Web Design Guidelines

Vercel React Native Skills

Vercel React Best Practices