[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7BHJDRXYNS9ZVFDTE20MMDP":3,"skill-same:01M7BHJDRXYNS9ZVFDTE20MMDP":82},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":41,"commitSha":42,"skillDirectory":43,"forksCount":44,"repositoryHtmlUrl":45,"repositoryHomepageUrl":46,"ownerHtmlUrl":47,"ownerBlogUrl":6,"ownerDescription":48,"updatedAt":49,"install":50,"resources":63},"01M7BHJDRXYNS9ZVFDTE20MMDP","huggingface-tokenizers",null,"Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in \u003C20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding\u002Ftruncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.","YYC-Cube\u002FYYC3-Knowledge-Base","YYC-Cube","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F235006543?v=4","01M79J2PH4SWD2X0825X6PN748","yyc3","01M79J2PHVK3F6ZBYT8K2SCZCW",1,"MIT","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-AI-应用工具\u002FYYC3-AI-应用工具-claude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Ftokenization-huggingface-tokenizers\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: huggingface-tokenizers\ndescription: Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in \u003C20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding\u002Ftruncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Tokenization, HuggingFace, BPE, WordPiece, Unigram, Fast Tokenization, Rust, Custom Tokenizer, Alignment Tracking, Production]\ndependencies: [tokenizers, transformers, datasets]\n---\n\n# HuggingFace Tokenizers - Fast Tokenization for NLP\n\nFast, production-ready tokenizers with Rust performance and Python ease-of-use.\n\n## When to use HuggingFace Tokenizers\n\n**Use HuggingFace Tokenizers when:**\n- Need extremely fast tokenization (\u003C20s per GB of text)\n- Training custom tokenizers from scratch\n- Want alignment tracking (token → original text position)\n- Building production NLP pipelines\n- Need to tokenize large corpora efficiently\n\n**Performance**:\n- **Speed**: \u003C20 seconds to tokenize 1GB on CPU\n- **Implementation**: Rust core with Python\u002FNode.js bindings\n- **Efficiency**: 10-100× faster than pure Python implementations\n\n**Use alternatives instead**:\n- **SentencePiece**: Language-independent, used by T5\u002FALBERT\n- **tiktoken**: OpenAI's BPE tokenizer for GPT models\n- **transformers AutoTokenizer**: Loading pretrained only (uses this library internally)\n\n## Quick start\n\n### Installation\n\n```bash\n# Install tokenizers\npip install tokenizers\n\n# With transformers integration\npip install tokenizers transformers\n```\n\n### Load pretrained tokenizer\n\n```python\nfrom tokenizers import Tokenizer\n\n# Load from HuggingFace Hub\ntokenizer = Tokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Encode text\noutput = tokenizer.encode(\"Hello, how are you?\")\nprint(output.tokens)  # ['hello', ',', 'how', 'are', 'you', '?']\nprint(output.ids)     # [7592, 1010, 2129, 2024, 2017, 1029]\n\n# Decode back\ntext = tokenizer.decode(output.ids)\nprint(text)  # \"hello, how are you?\"\n```\n\n### Train custom BPE tokenizer\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\n\n# Initialize tokenizer with BPE model\ntokenizer = Tokenizer(BPE(unk_token=\"[UNK]\"))\ntokenizer.pre_tokenizer = Whitespace()\n\n# Configure trainer\ntrainer = BpeTrainer(\n    vocab_size=30000,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    min_frequency=2\n)\n\n# Train on files\nfiles = [\"train.txt\", \"validation.txt\"]\ntokenizer.train(files, trainer)\n\n# Save\ntokenizer.save(\"my-tokenizer.json\")\n```\n\n**Training time**: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB\n\n### Batch encoding with padding\n\n```python\n# Enable padding\ntokenizer.enable_padding(pad_id=3, pad_token=\"[PAD]\")\n\n# Encode batch\ntexts = [\"Hello world\", \"This is a longer sentence\"]\nencodings = tokenizer.encode_batch(texts)\n\nfor encoding in encodings:\n    print(encoding.ids)\n# [101, 7592, 2088, 102, 3, 3, 3]\n# [101, 2023, 2003, 1037, 2936, 6251, 102]\n```\n\n## Tokenization algorithms\n\n### BPE (Byte-Pair Encoding)\n\n**How it works**:\n1. Start with character-level vocabulary\n2. Find most frequent character pair\n3. Merge into new token, add to vocabulary\n4. Repeat until vocabulary size reached\n\n**Used by**: GPT-2, GPT-3, RoBERTa, BART, DeBERTa\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import ByteLevel\n\ntokenizer = Tokenizer(BPE(unk_token=\"\u003C|endoftext|>\"))\ntokenizer.pre_tokenizer = ByteLevel()\n\ntrainer = BpeTrainer(\n    vocab_size=50257,\n    special_tokens=[\"\u003C|endoftext|>\"],\n    min_frequency=2\n)\n\ntokenizer.train(files=[\"data.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Handles OOV words well (breaks into subwords)\n- Flexible vocabulary size\n- Good for morphologically rich languages\n\n**Trade-offs**:\n- Tokenization depends on merge order\n- May split common words unexpectedly\n\n### WordPiece\n\n**How it works**:\n1. Start with character vocabulary\n2. Score merge pairs: `frequency(pair) \u002F (frequency(first) × frequency(second))`\n3. Merge highest scoring pair\n4. Repeat until vocabulary size reached\n\n**Used by**: BERT, DistilBERT, MobileBERT\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import WordPiece\nfrom tokenizers.trainers import WordPieceTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\nfrom tokenizers.normalizers import BertNormalizer\n\ntokenizer = Tokenizer(WordPiece(unk_token=\"[UNK]\"))\ntokenizer.normalizer = BertNormalizer(lowercase=True)\ntokenizer.pre_tokenizer = Whitespace()\n\ntrainer = WordPieceTrainer(\n    vocab_size=30522,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    continuing_subword_prefix=\"##\"\n)\n\ntokenizer.train(files=[\"corpus.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Prioritizes meaningful merges (high score = semantically related)\n- Used successfully in BERT (state-of-the-art results)\n\n**Trade-offs**:\n- Unknown words become `[UNK]` if no subword match\n- Saves vocabulary, not merge rules (larger files)\n\n### Unigram\n\n**How it works**:\n1. Start with large vocabulary (all substrings)\n2. Compute loss for corpus with current vocabulary\n3. Remove tokens with minimal impact on loss\n4. Repeat until vocabulary size reached\n\n**Used by**: ALBERT, T5, mBART, XLNet (via SentencePiece)\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import Unigram\nfrom tokenizers.trainers import UnigramTrainer\n\ntokenizer = Tokenizer(Unigram())\n\ntrainer = UnigramTrainer(\n    vocab_size=8000,\n    special_tokens=[\"\u003Cunk>\", \"\u003Cs>\", \"\u003C\u002Fs>\"],\n    unk_token=\"\u003Cunk>\"\n)\n\ntokenizer.train(files=[\"data.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Probabilistic (finds most likely tokenization)\n- Works well for languages without word boundaries\n- Handles diverse linguistic contexts\n\n**Trade-offs**:\n- Computationally expensive to train\n- More hyperparameters to tune\n\n## Tokenization pipeline\n\nComplete pipeline: **Normalization → Pre-tokenization → Model → Post-processing**\n\n### Normalization\n\nClean and standardize text:\n\n```python\nfrom tokenizers.normalizers import NFD, StripAccents, Lowercase, Sequence\n\ntokenizer.normalizer = Sequence([\n    NFD(),           # Unicode normalization (decompose)\n    Lowercase(),     # Convert to lowercase\n    StripAccents()   # Remove accents\n])\n\n# Input: \"Héllo WORLD\"\n# After normalization: \"hello world\"\n```\n\n**Common normalizers**:\n- `NFD`, `NFC`, `NFKD`, `NFKC` - Unicode normalization forms\n- `Lowercase()` - Convert to lowercase\n- `StripAccents()` - Remove accents (é → e)\n- `Strip()` - Remove whitespace\n- `Replace(pattern, content)` - Regex replacement\n\n### Pre-tokenization\n\nSplit text into word-like units:\n\n```python\nfrom tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence, ByteLevel\n\n# Split on whitespace and punctuation\ntokenizer.pre_tokenizer = Sequence([\n    Whitespace(),\n    Punctuation()\n])\n\n# Input: \"Hello, world!\"\n# After pre-tokenization: [\"Hello\", \",\", \"world\", \"!\"]\n```\n\n**Common pre-tokenizers**:\n- `Whitespace()` - Split on spaces, tabs, newlines\n- `ByteLevel()` - GPT-2 style byte-level splitting\n- `Punctuation()` - Isolate punctuation\n- `Digits(individual_digits=True)` - Split digits individually\n- `Metaspace()` - Replace spaces with ▁ (SentencePiece style)\n\n### Post-processing\n\nAdd special tokens for model input:\n\n```python\nfrom tokenizers.processors import TemplateProcessing\n\n# BERT-style: [CLS] sentence [SEP]\ntokenizer.post_processor = TemplateProcessing(\n    single=\"[CLS] $A [SEP]\",\n    pair=\"[CLS] $A [SEP] $B [SEP]\",\n    special_tokens=[\n        (\"[CLS]\", 1),\n        (\"[SEP]\", 2),\n    ],\n)\n```\n\n**Common patterns**:\n```python\n# GPT-2: sentence \u003C|endoftext|>\nTemplateProcessing(\n    single=\"$A \u003C|endoftext|>\",\n    special_tokens=[(\"\u003C|endoftext|>\", 50256)]\n)\n\n# RoBERTa: \u003Cs> sentence \u003C\u002Fs>\nTemplateProcessing(\n    single=\"\u003Cs> $A \u003C\u002Fs>\",\n    pair=\"\u003Cs> $A \u003C\u002Fs> \u003C\u002Fs> $B \u003C\u002Fs>\",\n    special_tokens=[(\"\u003Cs>\", 0), (\"\u003C\u002Fs>\", 2)]\n)\n```\n\n## Alignment tracking\n\nTrack token positions in original text:\n\n```python\noutput = tokenizer.encode(\"Hello, world!\")\n\n# Get token offsets\nfor token, offset in zip(output.tokens, output.offsets):\n    start, end = offset\n    print(f\"{token:10} → [{start:2}, {end:2}): {text[start:end]!r}\")\n\n# Output:\n# hello      → [ 0,  5): 'Hello'\n# ,          → [ 5,  6): ','\n# world      → [ 7, 12): 'world'\n# !          → [12, 13): '!'\n```\n\n**Use cases**:\n- Named entity recognition (map predictions back to text)\n- Question answering (extract answer spans)\n- Token classification (align labels to original positions)\n\n## Integration with transformers\n\n### Load with AutoTokenizer\n\n```python\nfrom transformers import AutoTokenizer\n\n# AutoTokenizer automatically uses fast tokenizers\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Check if using fast tokenizer\nprint(tokenizer.is_fast)  # True\n\n# Access underlying tokenizers.Tokenizer\nfast_tokenizer = tokenizer.backend_tokenizer\nprint(type(fast_tokenizer))  # \u003Cclass 'tokenizers.Tokenizer'>\n```\n\n### Convert custom tokenizer to transformers\n\n```python\nfrom tokenizers import Tokenizer\nfrom transformers import PreTrainedTokenizerFast\n\n# Train custom tokenizer\ntokenizer = Tokenizer(BPE())\n# ... train tokenizer ...\ntokenizer.save(\"my-tokenizer.json\")\n\n# Wrap for transformers\ntransformers_tokenizer = PreTrainedTokenizerFast(\n    tokenizer_file=\"my-tokenizer.json\",\n    unk_token=\"[UNK]\",\n    pad_token=\"[PAD]\",\n    cls_token=\"[CLS]\",\n    sep_token=\"[SEP]\",\n    mask_token=\"[MASK]\"\n)\n\n# Use like any transformers tokenizer\noutputs = transformers_tokenizer(\n    \"Hello world\",\n    padding=True,\n    truncation=True,\n    max_length=512,\n    return_tensors=\"pt\"\n)\n```\n\n## Common patterns\n\n### Train from iterator (large datasets)\n\n```python\nfrom datasets import load_dataset\n\n# Load dataset\ndataset = load_dataset(\"wikitext\", \"wikitext-103-raw-v1\", split=\"train\")\n\n# Create batch iterator\ndef batch_iterator(batch_size=1000):\n    for i in range(0, len(dataset), batch_size):\n        yield dataset[i:i + batch_size][\"text\"]\n\n# Train tokenizer\ntokenizer.train_from_iterator(\n    batch_iterator(),\n    trainer=trainer,\n    length=len(dataset)  # For progress bar\n)\n```\n\n**Performance**: Processes 1GB in ~10-20 minutes\n\n### Enable truncation and padding\n\n```python\n# Enable truncation\ntokenizer.enable_truncation(max_length=512)\n\n# Enable padding\ntokenizer.enable_padding(\n    pad_id=tokenizer.token_to_id(\"[PAD]\"),\n    pad_token=\"[PAD]\",\n    length=512  # Fixed length, or None for batch max\n)\n\n# Encode with both\noutput = tokenizer.encode(\"This is a long sentence that will be truncated...\")\nprint(len(output.ids))  # 512\n```\n\n### Multi-processing\n\n```python\nfrom tokenizers import Tokenizer\nfrom multiprocessing import Pool\n\n# Load tokenizer\ntokenizer = Tokenizer.from_file(\"tokenizer.json\")\n\ndef encode_batch(texts):\n    return tokenizer.encode_batch(texts)\n\n# Process large corpus in parallel\nwith Pool(8) as pool:\n    # Split corpus into chunks\n    chunk_size = 1000\n    chunks = [corpus[i:i+chunk_size] for i in range(0, len(corpus), chunk_size)]\n\n    # Encode in parallel\n    results = pool.map(encode_batch, chunks)\n```\n\n**Speedup**: 5-8× with 8 cores\n\n## Performance benchmarks\n\n### Training speed\n\n| Corpus Size | BPE (30k vocab) | WordPiece (30k) | Unigram (8k) |\n|-------------|-----------------|-----------------|--------------|\n| 10 MB       | 15 sec          | 18 sec          | 25 sec       |\n| 100 MB      | 1.5 min         | 2 min           | 4 min        |\n| 1 GB        | 15 min          | 20 min          | 40 min       |\n\n**Hardware**: 16-core CPU, tested on English Wikipedia\n\n### Tokenization speed\n\n| Implementation | 1 GB corpus | Throughput    |\n|----------------|-------------|---------------|\n| Pure Python    | ~20 minutes | ~50 MB\u002Fmin    |\n| HF Tokenizers  | ~15 seconds | ~4 GB\u002Fmin     |\n| **Speedup**    | **80×**     | **80×**       |\n\n**Test**: English text, average sentence length 20 words\n\n### Memory usage\n\n| Task                    | Memory  |\n|-------------------------|---------|\n| Load tokenizer          | ~10 MB  |\n| Train BPE (30k vocab)   | ~200 MB |\n| Encode 1M sentences     | ~500 MB |\n\n## Supported models\n\nPre-trained tokenizers available via `from_pretrained()`:\n\n**BERT family**:\n- `bert-base-uncased`, `bert-large-cased`\n- `distilbert-base-uncased`\n- `roberta-base`, `roberta-large`\n\n**GPT family**:\n- `gpt2`, `gpt2-medium`, `gpt2-large`\n- `distilgpt2`\n\n**T5 family**:\n- `t5-small`, `t5-base`, `t5-large`\n- `google\u002Fflan-t5-xxl`\n\n**Other**:\n- `facebook\u002Fbart-base`, `facebook\u002Fmbart-large-cc25`\n- `albert-base-v2`, `albert-xlarge-v2`\n- `xlm-roberta-base`, `xlm-roberta-large`\n\nBrowse all: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=tokenizers\n\n## References\n\n- **[Training Guide](references\u002Ftraining.md)** - Train custom tokenizers, configure trainers, handle large datasets\n- **[Algorithms Deep Dive](references\u002Falgorithms.md)** - BPE, WordPiece, Unigram explained in detail\n- **[Pipeline Components](references\u002Fpipeline.md)** - Normalizers, pre-tokenizers, post-processors, decoders\n- **[Transformers Integration](references\u002Fintegration.md)** - AutoTokenizer, PreTrainedTokenizerFast, special tokens\n\n## Resources\n\n- **Docs**: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftokenizers\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftokenizers ⭐ 9,000+\n- **Version**: 0.20.0+\n- **Course**: https:\u002F\u002Fhuggingface.co\u002Flearn\u002Fnlp-course\u002Fchapter6\u002F1\n- **Paper**: BPE (Sennrich et al., 2016), WordPiece (Schuster & Nakajima, 2012)\n\n\n","# HuggingFace Tokenizers - Fast Tokenization for NLP\n\nFast, production-ready tokenizers with Rust performance and Python ease-of-use.\n\n## When to use HuggingFace Tokenizers\n\n**Use HuggingFace Tokenizers when:**\n- Need extremely fast tokenization (\u003C20s per GB of text)\n- Training custom tokenizers from scratch\n- Want alignment tracking (token → original text position)\n- Building production NLP pipelines\n- Need to tokenize large corpora efficiently\n\n**Performance**:\n- **Speed**: \u003C20 seconds to tokenize 1GB on CPU\n- **Implementation**: Rust core with Python\u002FNode.js bindings\n- **Efficiency**: 10-100× faster than pure Python implementations\n\n**Use alternatives instead**:\n- **SentencePiece**: Language-independent, used by T5\u002FALBERT\n- **tiktoken**: OpenAI's BPE tokenizer for GPT models\n- **transformers AutoTokenizer**: Loading pretrained only (uses this library internally)\n\n## Quick start\n\n### Installation\n\n```bash\n# Install tokenizers\npip install tokenizers\n\n# With transformers integration\npip install tokenizers transformers\n```\n\n### Load pretrained tokenizer\n\n```python\nfrom tokenizers import Tokenizer\n\n# Load from HuggingFace Hub\ntokenizer = Tokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Encode text\noutput = tokenizer.encode(\"Hello, how are you?\")\nprint(output.tokens)  # ['hello', ',', 'how', 'are', 'you', '?']\nprint(output.ids)     # [7592, 1010, 2129, 2024, 2017, 1029]\n\n# Decode back\ntext = tokenizer.decode(output.ids)\nprint(text)  # \"hello, how are you?\"\n```\n\n### Train custom BPE tokenizer\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\n\n# Initialize tokenizer with BPE model\ntokenizer = Tokenizer(BPE(unk_token=\"[UNK]\"))\ntokenizer.pre_tokenizer = Whitespace()\n\n# Configure trainer\ntrainer = BpeTrainer(\n    vocab_size=30000,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    min_frequency=2\n)\n\n# Train on files\nfiles = [\"train.txt\", \"validation.txt\"]\ntokenizer.train(files, trainer)\n\n# Save\ntokenizer.save(\"my-tokenizer.json\")\n```\n\n**Training time**: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB\n\n### Batch encoding with padding\n\n```python\n# Enable padding\ntokenizer.enable_padding(pad_id=3, pad_token=\"[PAD]\")\n\n# Encode batch\ntexts = [\"Hello world\", \"This is a longer sentence\"]\nencodings = tokenizer.encode_batch(texts)\n\nfor encoding in encodings:\n    print(encoding.ids)\n# [101, 7592, 2088, 102, 3, 3, 3]\n# [101, 2023, 2003, 1037, 2936, 6251, 102]\n```\n\n## Tokenization algorithms\n\n### BPE (Byte-Pair Encoding)\n\n**How it works**:\n1. Start with character-level vocabulary\n2. Find most frequent character pair\n3. Merge into new token, add to vocabulary\n4. Repeat until vocabulary size reached\n\n**Used by**: GPT-2, GPT-3, RoBERTa, BART, DeBERTa\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import ByteLevel\n\ntokenizer = Tokenizer(BPE(unk_token=\"\u003C|endoftext|>\"))\ntokenizer.pre_tokenizer = ByteLevel()\n\ntrainer = BpeTrainer(\n    vocab_size=50257,\n    special_tokens=[\"\u003C|endoftext|>\"],\n    min_frequency=2\n)\n\ntokenizer.train(files=[\"data.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Handles OOV words well (breaks into subwords)\n- Flexible vocabulary size\n- Good for morphologically rich languages\n\n**Trade-offs**:\n- Tokenization depends on merge order\n- May split common words unexpectedly\n\n### WordPiece\n\n**How it works**:\n1. Start with character vocabulary\n2. Score merge pairs: `frequency(pair) \u002F (frequency(first) × frequency(second))`\n3. Merge highest scoring pair\n4. Repeat until vocabulary size reached\n\n**Used by**: BERT, DistilBERT, MobileBERT\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import WordPiece\nfrom tokenizers.trainers import WordPieceTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\nfrom tokenizers.normalizers import BertNormalizer\n\ntokenizer = Tokenizer(WordPiece(unk_token=\"[UNK]\"))\ntokenizer.normalizer = BertNormalizer(lowercase=True)\ntokenizer.pre_tokenizer = Whitespace()\n\ntrainer = WordPieceTrainer(\n    vocab_size=30522,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    continuing_subword_prefix=\"##\"\n)\n\ntokenizer.train(files=[\"corpus.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Prioritizes meaningful merges (high score = semantically related)\n- Used successfully in BERT (state-of-the-art results)\n\n**Trade-offs**:\n- Unknown words become `[UNK]` if no subword match\n- Saves vocabulary, not merge rules (larger files)\n\n### Unigram\n\n**How it works**:\n1. Start with large vocabulary (all substrings)\n2. Compute loss for corpus with current vocabulary\n3. Remove tokens with minimal impact on loss\n4. Repeat until vocabulary size reached\n\n**Used by**: ALBERT, T5, mBART, XLNet (via SentencePiece)\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import Unigram\nfrom tokenizers.trainers import UnigramTrainer\n\ntokenizer = Tokenizer(Unigram())\n\ntrainer = UnigramTrainer(\n    vocab_size=8000,\n    special_tokens=[\"\u003Cunk>\", \"\u003Cs>\", \"\u003C\u002Fs>\"],\n    unk_token=\"\u003Cunk>\"\n)\n\ntokenizer.train(files=[\"data.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Probabilistic (finds most likely tokenization)\n- Works well for languages without word boundaries\n- Handles diverse linguistic contexts\n\n**Trade-offs**:\n- Computationally expensive to train\n- More hyperparameters to tune\n\n## Tokenization pipeline\n\nComplete pipeline: **Normalization → Pre-tokenization → Model → Post-processing**\n\n### Normalization\n\nClean and standardize text:\n\n```python\nfrom tokenizers.normalizers import NFD, StripAccents, Lowercase, Sequence\n\ntokenizer.normalizer = Sequence([\n    NFD(),           # Unicode normalization (decompose)\n    Lowercase(),     # Convert to lowercase\n    StripAccents()   # Remove accents\n])\n\n# Input: \"Héllo WORLD\"\n# After normalization: \"hello world\"\n```\n\n**Common normalizers**:\n- `NFD`, `NFC`, `NFKD`, `NFKC` - Unicode normalization forms\n- `Lowercase()` - Convert to lowercase\n- `StripAccents()` - Remove accents (é → e)\n- `Strip()` - Remove whitespace\n- `Replace(pattern, content)` - Regex replacement\n\n### Pre-tokenization\n\nSplit text into word-like units:\n\n```python\nfrom tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence, ByteLevel\n\n# Split on whitespace and punctuation\ntokenizer.pre_tokenizer = Sequence([\n    Whitespace(),\n    Punctuation()\n])\n\n# Input: \"Hello, world!\"\n# After pre-tokenization: [\"Hello\", \",\", \"world\", \"!\"]\n```\n\n**Common pre-tokenizers**:\n- `Whitespace()` - Split on spaces, tabs, newlines\n- `ByteLevel()` - GPT-2 style byte-level splitting\n- `Punctuation()` - Isolate punctuation\n- `Digits(individual_digits=True)` - Split digits individually\n- `Metaspace()` - Replace spaces with ▁ (SentencePiece style)\n\n### Post-processing\n\nAdd special tokens for model input:\n\n```python\nfrom tokenizers.processors import TemplateProcessing\n\n# BERT-style: [CLS] sentence [SEP]\ntokenizer.post_processor = TemplateProcessing(\n    single=\"[CLS] $A [SEP]\",\n    pair=\"[CLS] $A [SEP] $B [SEP]\",\n    special_tokens=[\n        (\"[CLS]\", 1),\n        (\"[SEP]\", 2),\n    ],\n)\n```\n\n**Common patterns**:\n```python\n# GPT-2: sentence \u003C|endoftext|>\nTemplateProcessing(\n    single=\"$A \u003C|endoftext|>\",\n    special_tokens=[(\"\u003C|endoftext|>\", 50256)]\n)\n\n# RoBERTa: \u003Cs> sentence \u003C\u002Fs>\nTemplateProcessing(\n    single=\"\u003Cs> $A \u003C\u002Fs>\",\n    pair=\"\u003Cs> $A \u003C\u002Fs> \u003C\u002Fs> $B \u003C\u002Fs>\",\n    special_tokens=[(\"\u003Cs>\", 0), (\"\u003C\u002Fs>\", 2)]\n)\n```\n\n## Alignment tracking\n\nTrack token positions in original text:\n\n```python\noutput = tokenizer.encode(\"Hello, world!\")\n\n# Get token offsets\nfor token, offset in zip(output.tokens, output.offsets):\n    start, end = offset\n    print(f\"{token:10} → [{start:2}, {end:2}): {text[start:end]!r}\")\n\n# Output:\n# hello      → [ 0,  5): 'Hello'\n# ,          → [ 5,  6): ','\n# world      → [ 7, 12): 'world'\n# !          → [12, 13): '!'\n```\n\n**Use cases**:\n- Named entity recognition (map predictions back to text)\n- Question answering (extract answer spans)\n- Token classification (align labels to original positions)\n\n## Integration with transformers\n\n### Load with AutoTokenizer\n\n```python\nfrom transformers import AutoTokenizer\n\n# AutoTokenizer automatically uses fast tokenizers\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Check if using fast tokenizer\nprint(tokenizer.is_fast)  # True\n\n# Access underlying tokenizers.Tokenizer\nfast_tokenizer = tokenizer.backend_tokenizer\nprint(type(fast_tokenizer))  # \u003Cclass 'tokenizers.Tokenizer'>\n```\n\n### Convert custom tokenizer to transformers\n\n```python\nfrom tokenizers import Tokenizer\nfrom transformers import PreTrainedTokenizerFast\n\n# Train custom tokenizer\ntokenizer = Tokenizer(BPE())\n# ... train tokenizer ...\ntokenizer.save(\"my-tokenizer.json\")\n\n# Wrap for transformers\ntransformers_tokenizer = PreTrainedTokenizerFast(\n    tokenizer_file=\"my-tokenizer.json\",\n    unk_token=\"[UNK]\",\n    pad_token=\"[PAD]\",\n    cls_token=\"[CLS]\",\n    sep_token=\"[SEP]\",\n    mask_token=\"[MASK]\"\n)\n\n# Use like any transformers tokenizer\noutputs = transformers_tokenizer(\n    \"Hello world\",\n    padding=True,\n    truncation=True,\n    max_length=512,\n    return_tensors=\"pt\"\n)\n```\n\n## Common patterns\n\n### Train from iterator (large datasets)\n\n```python\nfrom datasets import load_dataset\n\n# Load dataset\ndataset = load_dataset(\"wikitext\", \"wikitext-103-raw-v1\", split=\"train\")\n\n# Create batch iterator\ndef batch_iterator(batch_size=1000):\n    for i in range(0, len(dataset), batch_size):\n        yield dataset[i:i + batch_size][\"text\"]\n\n# Train tokenizer\ntokenizer.train_from_iterator(\n    batch_iterator(),\n    trainer=trainer,\n    length=len(dataset)  # For progress bar\n)\n```\n\n**Performance**: Processes 1GB in ~10-20 minutes\n\n### Enable truncation and padding\n\n```python\n# Enable truncation\ntokenizer.enable_truncation(max_length=512)\n\n# Enable padding\ntokenizer.enable_padding(\n    pad_id=tokenizer.token_to_id(\"[PAD]\"),\n    pad_token=\"[PAD]\",\n    length=512  # Fixed length, or None for batch max\n)\n\n# Encode with both\noutput = tokenizer.encode(\"This is a long sentence that will be truncated...\")\nprint(len(output.ids))  # 512\n```\n\n### Multi-processing\n\n```python\nfrom tokenizers import Tokenizer\nfrom multiprocessing import Pool\n\n# Load tokenizer\ntokenizer = Tokenizer.from_file(\"tokenizer.json\")\n\ndef encode_batch(texts):\n    return tokenizer.encode_batch(texts)\n\n# Process large corpus in parallel\nwith Pool(8) as pool:\n    # Split corpus into chunks\n    chunk_size = 1000\n    chunks = [corpus[i:i+chunk_size] for i in range(0, len(corpus), chunk_size)]\n\n    # Encode in parallel\n    results = pool.map(encode_batch, chunks)\n```\n\n**Speedup**: 5-8× with 8 cores\n\n## Performance benchmarks\n\n### Training speed\n\n| Corpus Size | BPE (30k vocab) | WordPiece (30k) | Unigram (8k) |\n|-------------|-----------------|-----------------|--------------|\n| 10 MB       | 15 sec          | 18 sec          | 25 sec       |\n| 100 MB      | 1.5 min         | 2 min           | 4 min        |\n| 1 GB        | 15 min          | 20 min          | 40 min       |\n\n**Hardware**: 16-core CPU, tested on English Wikipedia\n\n### Tokenization speed\n\n| Implementation | 1 GB corpus | Throughput    |\n|----------------|-------------|---------------|\n| Pure Python    | ~20 minutes | ~50 MB\u002Fmin    |\n| HF Tokenizers  | ~15 seconds | ~4 GB\u002Fmin     |\n| **Speedup**    | **80×**     | **80×**       |\n\n**Test**: English text, average sentence length 20 words\n\n### Memory usage\n\n| Task                    | Memory  |\n|-------------------------|---------|\n| Load tokenizer          | ~10 MB  |\n| Train BPE (30k vocab)   | ~200 MB |\n| Encode 1M sentences     | ~500 MB |\n\n## Supported models\n\nPre-trained tokenizers available via `from_pretrained()`:\n\n**BERT family**:\n- `bert-base-uncased`, `bert-large-cased`\n- `distilbert-base-uncased`\n- `roberta-base`, `roberta-large`\n\n**GPT family**:\n- `gpt2`, `gpt2-medium`, `gpt2-large`\n- `distilgpt2`\n\n**T5 family**:\n- `t5-small`, `t5-base`, `t5-large`\n- `google\u002Fflan-t5-xxl`\n\n**Other**:\n- `facebook\u002Fbart-base`, `facebook\u002Fmbart-large-cc25`\n- `albert-base-v2`, `albert-xlarge-v2`\n- `xlm-roberta-base`, `xlm-roberta-large`\n\nBrowse all: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=tokenizers\n\n## References\n\n- **[Training Guide](references\u002Ftraining.md)** - Train custom tokenizers, configure trainers, handle large datasets\n- **[Algorithms Deep Dive](references\u002Falgorithms.md)** - BPE, WordPiece, Unigram explained in detail\n- **[Pipeline Components](references\u002Fpipeline.md)** - Normalizers, pre-tokenizers, post-processors, decoders\n- **[Transformers Integration](references\u002Fintegration.md)** - AutoTokenizer, PreTrainedTokenizerFast, special tokens\n\n## Resources\n\n- **Docs**: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftokenizers\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftokenizers ⭐ 9,000+\n- **Version**: 0.20.0+\n- **Course**: https:\u002F\u002Fhuggingface.co\u002Flearn\u002Fnlp-course\u002Fchapter6\u002F1\n- **Paper**: BPE (Sennrich et al., 2016), WordPiece (Schuster & Nakajima, 2012)\n\n\n",{"name":5,"tags":24,"author":35,"license":15,"version":36,"description":7,"dependencies":37},[25,26,27,28,29,30,31,32,33,34],"Tokenization","HuggingFace","BPE","WordPiece","Unigram","Fast Tokenization","Rust","Custom Tokenizer","Alignment Tracking","Production","Orchestra Research","1.0.0",[38,39,40],"tokenizers","transformers","datasets","main","87afe6ce96220accf50a345dea46784c8c9f210e","YYC3-AI-应用工具\u002FYYC3-AI-应用工具-claude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Ftokenization-huggingface-tokenizers",0,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base","https:\u002F\u002Fkb.yyc3.top\u002F","https:\u002F\u002Fgithub.com\u002FYYC-Cube","万象归元于云枢 | 深栈智启新纪元","2026-11-10 10:37:32",{"source":8,"skillName":5,"commands":51,"prompt":61,"zipAvailable":62},[52,55,58],{"tool":53,"command":54},"npx","npx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill huggingface-tokenizers",{"tool":56,"command":57},"bunx","bunx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill huggingface-tokenizers",{"tool":59,"command":60},"pnpm","pnpm dlx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill huggingface-tokenizers","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-AI-应用工具\u002FYYC3-AI-应用工具-claude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Ftokenization-huggingface-tokenizers\u002FSKILL.md",true,[64,70,74,78],{"path":65,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":68,"sourceHtmlUrl":69},"references\u002Falgorithms.md","REFERENCE","参考文档",15182,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-AI-应用工具\u002FYYC3-AI-应用工具-claude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Ftokenization-huggingface-tokenizers\u002Freferences\u002Falgorithms.md",{"path":71,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":72,"sourceHtmlUrl":73},"references\u002Fintegration.md",15389,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-AI-应用工具\u002FYYC3-AI-应用工具-claude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Ftokenization-huggingface-tokenizers\u002Freferences\u002Fintegration.md",{"path":75,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":76,"sourceHtmlUrl":77},"references\u002Fpipeline.md",16746,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-AI-应用工具\u002FYYC3-AI-应用工具-claude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Ftokenization-huggingface-tokenizers\u002Freferences\u002Fpipeline.md",{"path":79,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Ftraining.md",14559,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-AI-应用工具\u002FYYC3-AI-应用工具-claude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Ftokenization-huggingface-tokenizers\u002Freferences\u002Ftraining.md",[83,90,97,104,111,118],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":19},"01M7CTDWHERCXPQCP0K03A86AS","vercel-sandbox","Run agent-browser + Chrome inside Vercel Sandbox microVMs for browser automation from any Vercel-deployed app. Use when the user needs browser automation in a Vercel app (Next.js, SvelteKit, Nuxt, Remix, Astro, etc.), wants to run headless Chrome without binary size limits, needs persistent browser sessions across commands, or wants ephemeral isolated browser environments. Triggers include \"Vercel Sandbox browser\", \"microVM Chrome\", \"agent-browser in sandbox\", \"browser automation on Vercel\", or any task requiring Chrome in a Vercel Sandbox.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fvercel-sandbox\u002FSKILL.md",[],[],{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":94,"categories":95,"keywords":96,"publishedAt":19},"01M7CTD8D2BNJY54S65M47HEJA","slack","Interact with Slack workspaces using browser automation. Use when the user needs to check unread channels, navigate Slack, send messages, extract data, find information, search conversations, or automate any Slack task. Triggers include \"check my Slack\", \"what channels have unreads\", \"send a message to\", \"search Slack for\", \"extract from Slack\", \"find who said\", or any task requiring programmatic Slack interaction.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fslack\u002FSKILL.md",[],[],{"skillNo":98,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":101,"categories":102,"keywords":103,"publishedAt":19},"01M7CTCSSB8ZP3Q58C3YZDZK1W","electron","Automate Electron desktop apps (VS Code, Slack, Discord, Figma, Notion, Spotify, etc.) using agent-browser via Chrome DevTools Protocol. Use when the user needs to interact with an Electron app, automate a desktop app, connect to a running app, control a native app, or test an Electron application. Triggers include \"automate Slack app\", \"control VS Code\", \"interact with Discord app\", \"test this Electron app\", \"connect to desktop app\", or any task requiring automation of a native Electron application.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Felectron\u002FSKILL.md",[],[],{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":108,"categories":109,"keywords":110,"publishedAt":19},"01M7CTCBJY76688V4DCZGWFHGQ","dogfood","Systematically explore and test a web application to find bugs, UX issues, and other problems. Use when asked to \"dogfood\", \"QA\", \"exploratory test\", \"find issues\", \"bug hunt\", \"test this app\u002Fsite\u002Fplatform\", or review the quality of a web application. Produces a structured report with full reproduction evidence -- step-by-step screenshots, repro videos, and detailed repro steps for every issue -- so findings can be handed directly to the responsible teams.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fdogfood\u002FSKILL.md",[],[],{"skillNo":112,"name":113,"nameZh":6,"description":114,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":115,"categories":116,"keywords":117,"publishedAt":19},"01M7CTC4H0FFGVEAQR9PA4GA1A","agent-browser","Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to \"open a website\", \"fill out a form\", \"click a button\", \"take a screenshot\", \"scrape data from a page\", \"test this web app\", \"login to a site\", \"automate browser actions\", or any task requiring programmatic web interaction.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fagent-browser\u002FSKILL.md",[],[],{"skillNo":119,"name":120,"nameZh":6,"description":121,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":122,"categories":123,"keywords":124,"publishedAt":19},"01M7CTBKVFYBG78FDSABSDJTTQ","shadcn","Manages shadcn components and projects — adding, searching, fixing, debugging, styling, and composing UI, including chat interfaces. Provides project context, component docs, and usage examples. Applies when working with shadcn\u002Fui, component registries, presets, --preset codes, or any project with a components.json file. Also triggers for \"shadcn init\", \"create an app with --preset\", \"switch to --preset\", and for building a design system or applying a DESIGN.md or brand spec to shadcn\u002Fui.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-UI-界面组件\u002FYYC3-UI-界面组件-ui\u002Fskills\u002Fshadcn\u002FSKILL.md",[],[]]