[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M77XH4MS0PYB2CCX2WR36E3M":3,"skill-same:01M77XH4MS0PYB2CCX2WR36E3M":80},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":41,"commitSha":42,"skillDirectory":43,"forksCount":14,"repositoryHtmlUrl":44,"repositoryHomepageUrl":6,"ownerHtmlUrl":45,"ownerBlogUrl":6,"ownerDescription":46,"updatedAt":47,"install":48,"resources":61},"01M77XH4MS0PYB2CCX2WR36E3M","huggingface-tokenizers",null,"Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in \u003C20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding\u002Ftruncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.","math12345678\u002Fquanticle","math12345678","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F74202272?v=4","01M77XDK5Z6CNQM8T107PHDX1P","Smyan Reddy","01M77XDK6P25R6WF4RAGG08X9S",0,"MIT","https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F02-tokenization\u002Fhuggingface-tokenizers\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: huggingface-tokenizers\ndescription: Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in \u003C20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding\u002Ftruncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Tokenization, HuggingFace, BPE, WordPiece, Unigram, Fast Tokenization, Rust, Custom Tokenizer, Alignment Tracking, Production]\ndependencies: [tokenizers, transformers, datasets]\n---\n\n# HuggingFace Tokenizers - Fast Tokenization for NLP\n\nFast, production-ready tokenizers with Rust performance and Python ease-of-use.\n\n## When to use HuggingFace Tokenizers\n\n**Use HuggingFace Tokenizers when:**\n- Need extremely fast tokenization (\u003C20s per GB of text)\n- Training custom tokenizers from scratch\n- Want alignment tracking (token → original text position)\n- Building production NLP pipelines\n- Need to tokenize large corpora efficiently\n\n**Performance**:\n- **Speed**: \u003C20 seconds to tokenize 1GB on CPU\n- **Implementation**: Rust core with Python\u002FNode.js bindings\n- **Efficiency**: 10-100× faster than pure Python implementations\n\n**Use alternatives instead**:\n- **SentencePiece**: Language-independent, used by T5\u002FALBERT\n- **tiktoken**: OpenAI's BPE tokenizer for GPT models\n- **transformers AutoTokenizer**: Loading pretrained only (uses this library internally)\n\n## Quick start\n\n### Installation\n\n```bash\n# Install tokenizers\npip install tokenizers\n\n# With transformers integration\npip install tokenizers transformers\n```\n\n### Load pretrained tokenizer\n\n```python\nfrom tokenizers import Tokenizer\n\n# Load from HuggingFace Hub\ntokenizer = Tokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Encode text\noutput = tokenizer.encode(\"Hello, how are you?\")\nprint(output.tokens)  # ['hello', ',', 'how', 'are', 'you', '?']\nprint(output.ids)     # [7592, 1010, 2129, 2024, 2017, 1029]\n\n# Decode back\ntext = tokenizer.decode(output.ids)\nprint(text)  # \"hello, how are you?\"\n```\n\n### Train custom BPE tokenizer\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\n\n# Initialize tokenizer with BPE model\ntokenizer = Tokenizer(BPE(unk_token=\"[UNK]\"))\ntokenizer.pre_tokenizer = Whitespace()\n\n# Configure trainer\ntrainer = BpeTrainer(\n    vocab_size=30000,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    min_frequency=2\n)\n\n# Train on files\nfiles = [\"train.txt\", \"validation.txt\"]\ntokenizer.train(files, trainer)\n\n# Save\ntokenizer.save(\"my-tokenizer.json\")\n```\n\n**Training time**: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB\n\n### Batch encoding with padding\n\n```python\n# Enable padding\ntokenizer.enable_padding(pad_id=3, pad_token=\"[PAD]\")\n\n# Encode batch\ntexts = [\"Hello world\", \"This is a longer sentence\"]\nencodings = tokenizer.encode_batch(texts)\n\nfor encoding in encodings:\n    print(encoding.ids)\n# [101, 7592, 2088, 102, 3, 3, 3]\n# [101, 2023, 2003, 1037, 2936, 6251, 102]\n```\n\n## Tokenization algorithms\n\n### BPE (Byte-Pair Encoding)\n\n**How it works**:\n1. Start with character-level vocabulary\n2. Find most frequent character pair\n3. Merge into new token, add to vocabulary\n4. Repeat until vocabulary size reached\n\n**Used by**: GPT-2, GPT-3, RoBERTa, BART, DeBERTa\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import ByteLevel\n\ntokenizer = Tokenizer(BPE(unk_token=\"\u003C|endoftext|>\"))\ntokenizer.pre_tokenizer = ByteLevel()\n\ntrainer = BpeTrainer(\n    vocab_size=50257,\n    special_tokens=[\"\u003C|endoftext|>\"],\n    min_frequency=2\n)\n\ntokenizer.train(files=[\"data.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Handles OOV words well (breaks into subwords)\n- Flexible vocabulary size\n- Good for morphologically rich languages\n\n**Trade-offs**:\n- Tokenization depends on merge order\n- May split common words unexpectedly\n\n### WordPiece\n\n**How it works**:\n1. Start with character vocabulary\n2. Score merge pairs: `frequency(pair) \u002F (frequency(first) × frequency(second))`\n3. Merge highest scoring pair\n4. Repeat until vocabulary size reached\n\n**Used by**: BERT, DistilBERT, MobileBERT\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import WordPiece\nfrom tokenizers.trainers import WordPieceTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\nfrom tokenizers.normalizers import BertNormalizer\n\ntokenizer = Tokenizer(WordPiece(unk_token=\"[UNK]\"))\ntokenizer.normalizer = BertNormalizer(lowercase=True)\ntokenizer.pre_tokenizer = Whitespace()\n\ntrainer = WordPieceTrainer(\n    vocab_size=30522,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    continuing_subword_prefix=\"##\"\n)\n\ntokenizer.train(files=[\"corpus.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Prioritizes meaningful merges (high score = semantically related)\n- Used successfully in BERT (state-of-the-art results)\n\n**Trade-offs**:\n- Unknown words become `[UNK]` if no subword match\n- Saves vocabulary, not merge rules (larger files)\n\n### Unigram\n\n**How it works**:\n1. Start with large vocabulary (all substrings)\n2. Compute loss for corpus with current vocabulary\n3. Remove tokens with minimal impact on loss\n4. Repeat until vocabulary size reached\n\n**Used by**: ALBERT, T5, mBART, XLNet (via SentencePiece)\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import Unigram\nfrom tokenizers.trainers import UnigramTrainer\n\ntokenizer = Tokenizer(Unigram())\n\ntrainer = UnigramTrainer(\n    vocab_size=8000,\n    special_tokens=[\"\u003Cunk>\", \"\u003Cs>\", \"\u003C\u002Fs>\"],\n    unk_token=\"\u003Cunk>\"\n)\n\ntokenizer.train(files=[\"data.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Probabilistic (finds most likely tokenization)\n- Works well for languages without word boundaries\n- Handles diverse linguistic contexts\n\n**Trade-offs**:\n- Computationally expensive to train\n- More hyperparameters to tune\n\n## Tokenization pipeline\n\nComplete pipeline: **Normalization → Pre-tokenization → Model → Post-processing**\n\n### Normalization\n\nClean and standardize text:\n\n```python\nfrom tokenizers.normalizers import NFD, StripAccents, Lowercase, Sequence\n\ntokenizer.normalizer = Sequence([\n    NFD(),           # Unicode normalization (decompose)\n    Lowercase(),     # Convert to lowercase\n    StripAccents()   # Remove accents\n])\n\n# Input: \"Héllo WORLD\"\n# After normalization: \"hello world\"\n```\n\n**Common normalizers**:\n- `NFD`, `NFC`, `NFKD`, `NFKC` - Unicode normalization forms\n- `Lowercase()` - Convert to lowercase\n- `StripAccents()` - Remove accents (é → e)\n- `Strip()` - Remove whitespace\n- `Replace(pattern, content)` - Regex replacement\n\n### Pre-tokenization\n\nSplit text into word-like units:\n\n```python\nfrom tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence, ByteLevel\n\n# Split on whitespace and punctuation\ntokenizer.pre_tokenizer = Sequence([\n    Whitespace(),\n    Punctuation()\n])\n\n# Input: \"Hello, world!\"\n# After pre-tokenization: [\"Hello\", \",\", \"world\", \"!\"]\n```\n\n**Common pre-tokenizers**:\n- `Whitespace()` - Split on spaces, tabs, newlines\n- `ByteLevel()` - GPT-2 style byte-level splitting\n- `Punctuation()` - Isolate punctuation\n- `Digits(individual_digits=True)` - Split digits individually\n- `Metaspace()` - Replace spaces with ▁ (SentencePiece style)\n\n### Post-processing\n\nAdd special tokens for model input:\n\n```python\nfrom tokenizers.processors import TemplateProcessing\n\n# BERT-style: [CLS] sentence [SEP]\ntokenizer.post_processor = TemplateProcessing(\n    single=\"[CLS] $A [SEP]\",\n    pair=\"[CLS] $A [SEP] $B [SEP]\",\n    special_tokens=[\n        (\"[CLS]\", 1),\n        (\"[SEP]\", 2),\n    ],\n)\n```\n\n**Common patterns**:\n```python\n# GPT-2: sentence \u003C|endoftext|>\nTemplateProcessing(\n    single=\"$A \u003C|endoftext|>\",\n    special_tokens=[(\"\u003C|endoftext|>\", 50256)]\n)\n\n# RoBERTa: \u003Cs> sentence \u003C\u002Fs>\nTemplateProcessing(\n    single=\"\u003Cs> $A \u003C\u002Fs>\",\n    pair=\"\u003Cs> $A \u003C\u002Fs> \u003C\u002Fs> $B \u003C\u002Fs>\",\n    special_tokens=[(\"\u003Cs>\", 0), (\"\u003C\u002Fs>\", 2)]\n)\n```\n\n## Alignment tracking\n\nTrack token positions in original text:\n\n```python\noutput = tokenizer.encode(\"Hello, world!\")\n\n# Get token offsets\nfor token, offset in zip(output.tokens, output.offsets):\n    start, end = offset\n    print(f\"{token:10} → [{start:2}, {end:2}): {text[start:end]!r}\")\n\n# Output:\n# hello      → [ 0,  5): 'Hello'\n# ,          → [ 5,  6): ','\n# world      → [ 7, 12): 'world'\n# !          → [12, 13): '!'\n```\n\n**Use cases**:\n- Named entity recognition (map predictions back to text)\n- Question answering (extract answer spans)\n- Token classification (align labels to original positions)\n\n## Integration with transformers\n\n### Load with AutoTokenizer\n\n```python\nfrom transformers import AutoTokenizer\n\n# AutoTokenizer automatically uses fast tokenizers\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Check if using fast tokenizer\nprint(tokenizer.is_fast)  # True\n\n# Access underlying tokenizers.Tokenizer\nfast_tokenizer = tokenizer.backend_tokenizer\nprint(type(fast_tokenizer))  # \u003Cclass 'tokenizers.Tokenizer'>\n```\n\n### Convert custom tokenizer to transformers\n\n```python\nfrom tokenizers import Tokenizer\nfrom transformers import PreTrainedTokenizerFast\n\n# Train custom tokenizer\ntokenizer = Tokenizer(BPE())\n# ... train tokenizer ...\ntokenizer.save(\"my-tokenizer.json\")\n\n# Wrap for transformers\ntransformers_tokenizer = PreTrainedTokenizerFast(\n    tokenizer_file=\"my-tokenizer.json\",\n    unk_token=\"[UNK]\",\n    pad_token=\"[PAD]\",\n    cls_token=\"[CLS]\",\n    sep_token=\"[SEP]\",\n    mask_token=\"[MASK]\"\n)\n\n# Use like any transformers tokenizer\noutputs = transformers_tokenizer(\n    \"Hello world\",\n    padding=True,\n    truncation=True,\n    max_length=512,\n    return_tensors=\"pt\"\n)\n```\n\n## Common patterns\n\n### Train from iterator (large datasets)\n\n```python\nfrom datasets import load_dataset\n\n# Load dataset\ndataset = load_dataset(\"wikitext\", \"wikitext-103-raw-v1\", split=\"train\")\n\n# Create batch iterator\ndef batch_iterator(batch_size=1000):\n    for i in range(0, len(dataset), batch_size):\n        yield dataset[i:i + batch_size][\"text\"]\n\n# Train tokenizer\ntokenizer.train_from_iterator(\n    batch_iterator(),\n    trainer=trainer,\n    length=len(dataset)  # For progress bar\n)\n```\n\n**Performance**: Processes 1GB in ~10-20 minutes\n\n### Enable truncation and padding\n\n```python\n# Enable truncation\ntokenizer.enable_truncation(max_length=512)\n\n# Enable padding\ntokenizer.enable_padding(\n    pad_id=tokenizer.token_to_id(\"[PAD]\"),\n    pad_token=\"[PAD]\",\n    length=512  # Fixed length, or None for batch max\n)\n\n# Encode with both\noutput = tokenizer.encode(\"This is a long sentence that will be truncated...\")\nprint(len(output.ids))  # 512\n```\n\n### Multi-processing\n\n```python\nfrom tokenizers import Tokenizer\nfrom multiprocessing import Pool\n\n# Load tokenizer\ntokenizer = Tokenizer.from_file(\"tokenizer.json\")\n\ndef encode_batch(texts):\n    return tokenizer.encode_batch(texts)\n\n# Process large corpus in parallel\nwith Pool(8) as pool:\n    # Split corpus into chunks\n    chunk_size = 1000\n    chunks = [corpus[i:i+chunk_size] for i in range(0, len(corpus), chunk_size)]\n\n    # Encode in parallel\n    results = pool.map(encode_batch, chunks)\n```\n\n**Speedup**: 5-8× with 8 cores\n\n## Performance benchmarks\n\n### Training speed\n\n| Corpus Size | BPE (30k vocab) | WordPiece (30k) | Unigram (8k) |\n|-------------|-----------------|-----------------|--------------|\n| 10 MB       | 15 sec          | 18 sec          | 25 sec       |\n| 100 MB      | 1.5 min         | 2 min           | 4 min        |\n| 1 GB        | 15 min          | 20 min          | 40 min       |\n\n**Hardware**: 16-core CPU, tested on English Wikipedia\n\n### Tokenization speed\n\n| Implementation | 1 GB corpus | Throughput    |\n|----------------|-------------|---------------|\n| Pure Python    | ~20 minutes | ~50 MB\u002Fmin    |\n| HF Tokenizers  | ~15 seconds | ~4 GB\u002Fmin     |\n| **Speedup**    | **80×**     | **80×**       |\n\n**Test**: English text, average sentence length 20 words\n\n### Memory usage\n\n| Task                    | Memory  |\n|-------------------------|---------|\n| Load tokenizer          | ~10 MB  |\n| Train BPE (30k vocab)   | ~200 MB |\n| Encode 1M sentences     | ~500 MB |\n\n## Supported models\n\nPre-trained tokenizers available via `from_pretrained()`:\n\n**BERT family**:\n- `bert-base-uncased`, `bert-large-cased`\n- `distilbert-base-uncased`\n- `roberta-base`, `roberta-large`\n\n**GPT family**:\n- `gpt2`, `gpt2-medium`, `gpt2-large`\n- `distilgpt2`\n\n**T5 family**:\n- `t5-small`, `t5-base`, `t5-large`\n- `google\u002Fflan-t5-xxl`\n\n**Other**:\n- `facebook\u002Fbart-base`, `facebook\u002Fmbart-large-cc25`\n- `albert-base-v2`, `albert-xlarge-v2`\n- `xlm-roberta-base`, `xlm-roberta-large`\n\nBrowse all: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=tokenizers\n\n## References\n\n- **[Training Guide](references\u002Ftraining.md)** - Train custom tokenizers, configure trainers, handle large datasets\n- **[Algorithms Deep Dive](references\u002Falgorithms.md)** - BPE, WordPiece, Unigram explained in detail\n- **[Pipeline Components](references\u002Fpipeline.md)** - Normalizers, pre-tokenizers, post-processors, decoders\n- **[Transformers Integration](references\u002Fintegration.md)** - AutoTokenizer, PreTrainedTokenizerFast, special tokens\n\n## Resources\n\n- **Docs**: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftokenizers\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftokenizers ⭐ 9,000+\n- **Version**: 0.20.0+\n- **Course**: https:\u002F\u002Fhuggingface.co\u002Flearn\u002Fnlp-course\u002Fchapter6\u002F1\n- **Paper**: BPE (Sennrich et al., 2016), WordPiece (Schuster & Nakajima, 2012)\n\n\n","# HuggingFace Tokenizers - Fast Tokenization for NLP\n\nFast, production-ready tokenizers with Rust performance and Python ease-of-use.\n\n## When to use HuggingFace Tokenizers\n\n**Use HuggingFace Tokenizers when:**\n- Need extremely fast tokenization (\u003C20s per GB of text)\n- Training custom tokenizers from scratch\n- Want alignment tracking (token → original text position)\n- Building production NLP pipelines\n- Need to tokenize large corpora efficiently\n\n**Performance**:\n- **Speed**: \u003C20 seconds to tokenize 1GB on CPU\n- **Implementation**: Rust core with Python\u002FNode.js bindings\n- **Efficiency**: 10-100× faster than pure Python implementations\n\n**Use alternatives instead**:\n- **SentencePiece**: Language-independent, used by T5\u002FALBERT\n- **tiktoken**: OpenAI's BPE tokenizer for GPT models\n- **transformers AutoTokenizer**: Loading pretrained only (uses this library internally)\n\n## Quick start\n\n### Installation\n\n```bash\n# Install tokenizers\npip install tokenizers\n\n# With transformers integration\npip install tokenizers transformers\n```\n\n### Load pretrained tokenizer\n\n```python\nfrom tokenizers import Tokenizer\n\n# Load from HuggingFace Hub\ntokenizer = Tokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Encode text\noutput = tokenizer.encode(\"Hello, how are you?\")\nprint(output.tokens)  # ['hello', ',', 'how', 'are', 'you', '?']\nprint(output.ids)     # [7592, 1010, 2129, 2024, 2017, 1029]\n\n# Decode back\ntext = tokenizer.decode(output.ids)\nprint(text)  # \"hello, how are you?\"\n```\n\n### Train custom BPE tokenizer\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\n\n# Initialize tokenizer with BPE model\ntokenizer = Tokenizer(BPE(unk_token=\"[UNK]\"))\ntokenizer.pre_tokenizer = Whitespace()\n\n# Configure trainer\ntrainer = BpeTrainer(\n    vocab_size=30000,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    min_frequency=2\n)\n\n# Train on files\nfiles = [\"train.txt\", \"validation.txt\"]\ntokenizer.train(files, trainer)\n\n# Save\ntokenizer.save(\"my-tokenizer.json\")\n```\n\n**Training time**: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB\n\n### Batch encoding with padding\n\n```python\n# Enable padding\ntokenizer.enable_padding(pad_id=3, pad_token=\"[PAD]\")\n\n# Encode batch\ntexts = [\"Hello world\", \"This is a longer sentence\"]\nencodings = tokenizer.encode_batch(texts)\n\nfor encoding in encodings:\n    print(encoding.ids)\n# [101, 7592, 2088, 102, 3, 3, 3]\n# [101, 2023, 2003, 1037, 2936, 6251, 102]\n```\n\n## Tokenization algorithms\n\n### BPE (Byte-Pair Encoding)\n\n**How it works**:\n1. Start with character-level vocabulary\n2. Find most frequent character pair\n3. Merge into new token, add to vocabulary\n4. Repeat until vocabulary size reached\n\n**Used by**: GPT-2, GPT-3, RoBERTa, BART, DeBERTa\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import ByteLevel\n\ntokenizer = Tokenizer(BPE(unk_token=\"\u003C|endoftext|>\"))\ntokenizer.pre_tokenizer = ByteLevel()\n\ntrainer = BpeTrainer(\n    vocab_size=50257,\n    special_tokens=[\"\u003C|endoftext|>\"],\n    min_frequency=2\n)\n\ntokenizer.train(files=[\"data.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Handles OOV words well (breaks into subwords)\n- Flexible vocabulary size\n- Good for morphologically rich languages\n\n**Trade-offs**:\n- Tokenization depends on merge order\n- May split common words unexpectedly\n\n### WordPiece\n\n**How it works**:\n1. Start with character vocabulary\n2. Score merge pairs: `frequency(pair) \u002F (frequency(first) × frequency(second))`\n3. Merge highest scoring pair\n4. Repeat until vocabulary size reached\n\n**Used by**: BERT, DistilBERT, MobileBERT\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import WordPiece\nfrom tokenizers.trainers import WordPieceTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\nfrom tokenizers.normalizers import BertNormalizer\n\ntokenizer = Tokenizer(WordPiece(unk_token=\"[UNK]\"))\ntokenizer.normalizer = BertNormalizer(lowercase=True)\ntokenizer.pre_tokenizer = Whitespace()\n\ntrainer = WordPieceTrainer(\n    vocab_size=30522,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    continuing_subword_prefix=\"##\"\n)\n\ntokenizer.train(files=[\"corpus.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Prioritizes meaningful merges (high score = semantically related)\n- Used successfully in BERT (state-of-the-art results)\n\n**Trade-offs**:\n- Unknown words become `[UNK]` if no subword match\n- Saves vocabulary, not merge rules (larger files)\n\n### Unigram\n\n**How it works**:\n1. Start with large vocabulary (all substrings)\n2. Compute loss for corpus with current vocabulary\n3. Remove tokens with minimal impact on loss\n4. Repeat until vocabulary size reached\n\n**Used by**: ALBERT, T5, mBART, XLNet (via SentencePiece)\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import Unigram\nfrom tokenizers.trainers import UnigramTrainer\n\ntokenizer = Tokenizer(Unigram())\n\ntrainer = UnigramTrainer(\n    vocab_size=8000,\n    special_tokens=[\"\u003Cunk>\", \"\u003Cs>\", \"\u003C\u002Fs>\"],\n    unk_token=\"\u003Cunk>\"\n)\n\ntokenizer.train(files=[\"data.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Probabilistic (finds most likely tokenization)\n- Works well for languages without word boundaries\n- Handles diverse linguistic contexts\n\n**Trade-offs**:\n- Computationally expensive to train\n- More hyperparameters to tune\n\n## Tokenization pipeline\n\nComplete pipeline: **Normalization → Pre-tokenization → Model → Post-processing**\n\n### Normalization\n\nClean and standardize text:\n\n```python\nfrom tokenizers.normalizers import NFD, StripAccents, Lowercase, Sequence\n\ntokenizer.normalizer = Sequence([\n    NFD(),           # Unicode normalization (decompose)\n    Lowercase(),     # Convert to lowercase\n    StripAccents()   # Remove accents\n])\n\n# Input: \"Héllo WORLD\"\n# After normalization: \"hello world\"\n```\n\n**Common normalizers**:\n- `NFD`, `NFC`, `NFKD`, `NFKC` - Unicode normalization forms\n- `Lowercase()` - Convert to lowercase\n- `StripAccents()` - Remove accents (é → e)\n- `Strip()` - Remove whitespace\n- `Replace(pattern, content)` - Regex replacement\n\n### Pre-tokenization\n\nSplit text into word-like units:\n\n```python\nfrom tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence, ByteLevel\n\n# Split on whitespace and punctuation\ntokenizer.pre_tokenizer = Sequence([\n    Whitespace(),\n    Punctuation()\n])\n\n# Input: \"Hello, world!\"\n# After pre-tokenization: [\"Hello\", \",\", \"world\", \"!\"]\n```\n\n**Common pre-tokenizers**:\n- `Whitespace()` - Split on spaces, tabs, newlines\n- `ByteLevel()` - GPT-2 style byte-level splitting\n- `Punctuation()` - Isolate punctuation\n- `Digits(individual_digits=True)` - Split digits individually\n- `Metaspace()` - Replace spaces with ▁ (SentencePiece style)\n\n### Post-processing\n\nAdd special tokens for model input:\n\n```python\nfrom tokenizers.processors import TemplateProcessing\n\n# BERT-style: [CLS] sentence [SEP]\ntokenizer.post_processor = TemplateProcessing(\n    single=\"[CLS] $A [SEP]\",\n    pair=\"[CLS] $A [SEP] $B [SEP]\",\n    special_tokens=[\n        (\"[CLS]\", 1),\n        (\"[SEP]\", 2),\n    ],\n)\n```\n\n**Common patterns**:\n```python\n# GPT-2: sentence \u003C|endoftext|>\nTemplateProcessing(\n    single=\"$A \u003C|endoftext|>\",\n    special_tokens=[(\"\u003C|endoftext|>\", 50256)]\n)\n\n# RoBERTa: \u003Cs> sentence \u003C\u002Fs>\nTemplateProcessing(\n    single=\"\u003Cs> $A \u003C\u002Fs>\",\n    pair=\"\u003Cs> $A \u003C\u002Fs> \u003C\u002Fs> $B \u003C\u002Fs>\",\n    special_tokens=[(\"\u003Cs>\", 0), (\"\u003C\u002Fs>\", 2)]\n)\n```\n\n## Alignment tracking\n\nTrack token positions in original text:\n\n```python\noutput = tokenizer.encode(\"Hello, world!\")\n\n# Get token offsets\nfor token, offset in zip(output.tokens, output.offsets):\n    start, end = offset\n    print(f\"{token:10} → [{start:2}, {end:2}): {text[start:end]!r}\")\n\n# Output:\n# hello      → [ 0,  5): 'Hello'\n# ,          → [ 5,  6): ','\n# world      → [ 7, 12): 'world'\n# !          → [12, 13): '!'\n```\n\n**Use cases**:\n- Named entity recognition (map predictions back to text)\n- Question answering (extract answer spans)\n- Token classification (align labels to original positions)\n\n## Integration with transformers\n\n### Load with AutoTokenizer\n\n```python\nfrom transformers import AutoTokenizer\n\n# AutoTokenizer automatically uses fast tokenizers\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Check if using fast tokenizer\nprint(tokenizer.is_fast)  # True\n\n# Access underlying tokenizers.Tokenizer\nfast_tokenizer = tokenizer.backend_tokenizer\nprint(type(fast_tokenizer))  # \u003Cclass 'tokenizers.Tokenizer'>\n```\n\n### Convert custom tokenizer to transformers\n\n```python\nfrom tokenizers import Tokenizer\nfrom transformers import PreTrainedTokenizerFast\n\n# Train custom tokenizer\ntokenizer = Tokenizer(BPE())\n# ... train tokenizer ...\ntokenizer.save(\"my-tokenizer.json\")\n\n# Wrap for transformers\ntransformers_tokenizer = PreTrainedTokenizerFast(\n    tokenizer_file=\"my-tokenizer.json\",\n    unk_token=\"[UNK]\",\n    pad_token=\"[PAD]\",\n    cls_token=\"[CLS]\",\n    sep_token=\"[SEP]\",\n    mask_token=\"[MASK]\"\n)\n\n# Use like any transformers tokenizer\noutputs = transformers_tokenizer(\n    \"Hello world\",\n    padding=True,\n    truncation=True,\n    max_length=512,\n    return_tensors=\"pt\"\n)\n```\n\n## Common patterns\n\n### Train from iterator (large datasets)\n\n```python\nfrom datasets import load_dataset\n\n# Load dataset\ndataset = load_dataset(\"wikitext\", \"wikitext-103-raw-v1\", split=\"train\")\n\n# Create batch iterator\ndef batch_iterator(batch_size=1000):\n    for i in range(0, len(dataset), batch_size):\n        yield dataset[i:i + batch_size][\"text\"]\n\n# Train tokenizer\ntokenizer.train_from_iterator(\n    batch_iterator(),\n    trainer=trainer,\n    length=len(dataset)  # For progress bar\n)\n```\n\n**Performance**: Processes 1GB in ~10-20 minutes\n\n### Enable truncation and padding\n\n```python\n# Enable truncation\ntokenizer.enable_truncation(max_length=512)\n\n# Enable padding\ntokenizer.enable_padding(\n    pad_id=tokenizer.token_to_id(\"[PAD]\"),\n    pad_token=\"[PAD]\",\n    length=512  # Fixed length, or None for batch max\n)\n\n# Encode with both\noutput = tokenizer.encode(\"This is a long sentence that will be truncated...\")\nprint(len(output.ids))  # 512\n```\n\n### Multi-processing\n\n```python\nfrom tokenizers import Tokenizer\nfrom multiprocessing import Pool\n\n# Load tokenizer\ntokenizer = Tokenizer.from_file(\"tokenizer.json\")\n\ndef encode_batch(texts):\n    return tokenizer.encode_batch(texts)\n\n# Process large corpus in parallel\nwith Pool(8) as pool:\n    # Split corpus into chunks\n    chunk_size = 1000\n    chunks = [corpus[i:i+chunk_size] for i in range(0, len(corpus), chunk_size)]\n\n    # Encode in parallel\n    results = pool.map(encode_batch, chunks)\n```\n\n**Speedup**: 5-8× with 8 cores\n\n## Performance benchmarks\n\n### Training speed\n\n| Corpus Size | BPE (30k vocab) | WordPiece (30k) | Unigram (8k) |\n|-------------|-----------------|-----------------|--------------|\n| 10 MB       | 15 sec          | 18 sec          | 25 sec       |\n| 100 MB      | 1.5 min         | 2 min           | 4 min        |\n| 1 GB        | 15 min          | 20 min          | 40 min       |\n\n**Hardware**: 16-core CPU, tested on English Wikipedia\n\n### Tokenization speed\n\n| Implementation | 1 GB corpus | Throughput    |\n|----------------|-------------|---------------|\n| Pure Python    | ~20 minutes | ~50 MB\u002Fmin    |\n| HF Tokenizers  | ~15 seconds | ~4 GB\u002Fmin     |\n| **Speedup**    | **80×**     | **80×**       |\n\n**Test**: English text, average sentence length 20 words\n\n### Memory usage\n\n| Task                    | Memory  |\n|-------------------------|---------|\n| Load tokenizer          | ~10 MB  |\n| Train BPE (30k vocab)   | ~200 MB |\n| Encode 1M sentences     | ~500 MB |\n\n## Supported models\n\nPre-trained tokenizers available via `from_pretrained()`:\n\n**BERT family**:\n- `bert-base-uncased`, `bert-large-cased`\n- `distilbert-base-uncased`\n- `roberta-base`, `roberta-large`\n\n**GPT family**:\n- `gpt2`, `gpt2-medium`, `gpt2-large`\n- `distilgpt2`\n\n**T5 family**:\n- `t5-small`, `t5-base`, `t5-large`\n- `google\u002Fflan-t5-xxl`\n\n**Other**:\n- `facebook\u002Fbart-base`, `facebook\u002Fmbart-large-cc25`\n- `albert-base-v2`, `albert-xlarge-v2`\n- `xlm-roberta-base`, `xlm-roberta-large`\n\nBrowse all: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=tokenizers\n\n## References\n\n- **[Training Guide](references\u002Ftraining.md)** - Train custom tokenizers, configure trainers, handle large datasets\n- **[Algorithms Deep Dive](references\u002Falgorithms.md)** - BPE, WordPiece, Unigram explained in detail\n- **[Pipeline Components](references\u002Fpipeline.md)** - Normalizers, pre-tokenizers, post-processors, decoders\n- **[Transformers Integration](references\u002Fintegration.md)** - AutoTokenizer, PreTrainedTokenizerFast, special tokens\n\n## Resources\n\n- **Docs**: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftokenizers\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftokenizers ⭐ 9,000+\n- **Version**: 0.20.0+\n- **Course**: https:\u002F\u002Fhuggingface.co\u002Flearn\u002Fnlp-course\u002Fchapter6\u002F1\n- **Paper**: BPE (Sennrich et al., 2016), WordPiece (Schuster & Nakajima, 2012)\n\n\n",{"name":5,"tags":24,"author":35,"license":15,"version":36,"description":7,"dependencies":37},[25,26,27,28,29,30,31,32,33,34],"Tokenization","HuggingFace","BPE","WordPiece","Unigram","Fast Tokenization","Rust","Custom Tokenizer","Alignment Tracking","Production","Orchestra Research","1.0.0",[38,39,40],"tokenizers","transformers","datasets","main","3c55f1641d1423e5271b321e2fd40d3a5ed92fd2","quanticlee\u002Fsrc\u002Fskills\u002F02-tokenization\u002Fhuggingface-tokenizers","https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle","https:\u002F\u002Fgithub.com\u002Fmath12345678","I'm a 10th-grade student working at the intersection of computational biology, molecular virology, and AI-driven approaches to medicine and global health.","2026-06-30 15:02:45",{"source":8,"skillName":5,"commands":49,"prompt":59,"zipAvailable":60},[50,53,56],{"tool":51,"command":52},"npx","npx skills add math12345678\u002Fquanticle --skill huggingface-tokenizers",{"tool":54,"command":55},"bunx","bunx skills add math12345678\u002Fquanticle --skill huggingface-tokenizers",{"tool":57,"command":58},"pnpm","pnpm dlx skills add math12345678\u002Fquanticle --skill huggingface-tokenizers","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F02-tokenization\u002Fhuggingface-tokenizers\u002FSKILL.md",true,[62,68,72,76],{"path":63,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":66,"sourceHtmlUrl":67},"references\u002Falgorithms.md","REFERENCE","参考文档",15182,"https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F02-tokenization\u002Fhuggingface-tokenizers\u002Freferences\u002Falgorithms.md",{"path":69,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":70,"sourceHtmlUrl":71},"references\u002Fintegration.md",15389,"https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F02-tokenization\u002Fhuggingface-tokenizers\u002Freferences\u002Fintegration.md",{"path":73,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":74,"sourceHtmlUrl":75},"references\u002Fpipeline.md",16746,"https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F02-tokenization\u002Fhuggingface-tokenizers\u002Freferences\u002Fpipeline.md",{"path":77,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":78,"sourceHtmlUrl":79},"references\u002Ftraining.md",14559,"https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F02-tokenization\u002Fhuggingface-tokenizers\u002Freferences\u002Ftraining.md",[81,88,95,102,109,116],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M77XYA93N9NZCDQQ58SCKQ49","ara-rigor-reviewer","Performs ARA Seal Level 2 semantic epistemic review on Agent-Native Research Artifacts, scoring six dimensions (evidence relevance, falsifiability, scope calibration, argument coherence, exploration integrity, methodological rigor) and producing a constructive, severity-ranked report with a Strong Accept-to-Reject recommendation. Use after Level 1 structural validation passes, when an ARA needs an objective epistemic critique before publication or release.","https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F22-agent-native-research-artifact\u002Frigor-reviewer\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M77XY5PY6DRG8YBZG24WQ039","ara-research-manager","Records research provenance as a post-task epilogue, scanning conversation history at the end of a coding or research session to extract decisions, experiments, dead ends, claims, heuristics, and pivots, and writing them into the ara\u002F directory with user-vs-AI provenance tags. Use as a session epilogue — never during execution — to maintain a faithful, auditable trace of how a research project actually evolved.","https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F22-agent-native-research-artifact\u002Fresearch-manager\u002FSKILL.md",[],[],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":19},"01M77XY138ZMY0JG84569T8MSW","ara-compiler","Compiles any research input — PDF papers, GitHub repositories, experiment logs, code directories, or raw notes — into a complete Agent-Native Research Artifact (ARA) with cognitive layer (claims, concepts, heuristics), physical layer (configs, code stubs), exploration graph, and grounded evidence. Use when ingesting a paper or codebase into a structured, machine-executable knowledge package, building an ARA from scratch, or converting research outputs into a falsifiable, agent-traversable form.","https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F22-agent-native-research-artifact\u002Fcompiler\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":19},"01M77XXVQYZ3VVHM9KFFKB0PDB","creative-thinking-for-research","Applies cognitive science frameworks for creative thinking to CS and AI research ideation. Use when seeking genuinely novel research directions by leveraging combinatorial creativity, analogical reasoning, constraint manipulation, and other empirically grounded creative strategies.","https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F21-research-ideation\u002Fcreative-thinking-for-research\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":19},"01M77XXPGJ9FR86MNBG74RKV3W","brainstorming-research-ideas","Guides researchers through structured ideation frameworks to discover high-impact research directions. Use when exploring new problem spaces, pivoting between projects, or seeking novel angles on existing work.","https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F21-research-ideation\u002Fbrainstorming-research-ideas\u002FSKILL.md",[],[],{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":19},"01M77XXG6R4E97DG3M1CT7CT0Q","systems-paper-writing","Comprehensive guide for writing systems papers targeting OSDI, SOSP, ASPLOS, NSDI, and EuroSys. Provides paragraph-level structural blueprints, writing patterns, venue-specific checklists, reviewer guidelines, LaTeX templates, and conference deadlines. Use this skill for all systems conference paper writing.","https:\u002F\u002Fgithub.com\u002Fmath12345678\u002Fquanticle\u002Fblob\u002Fmain\u002Fquanticlee\u002Fsrc\u002Fskills\u002F20-ml-paper-writing\u002Fsystems-paper-writing\u002FSKILL.md",[],[]]