[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MA8DN63PD84QDAJZB71D8D":3,"skill-same:01M6MA8DN63PD84QDAJZB71D8D":98},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":41,"linkOnly":42,"markdownContent":43,"markdownBody":44,"frontMatter":45,"refName":52,"commitSha":53,"skillDirectory":54,"forksCount":55,"repositoryHtmlUrl":56,"repositoryHomepageUrl":57,"ownerHtmlUrl":58,"ownerBlogUrl":59,"ownerDescription":60,"updatedAt":61,"install":62,"resources":75},"01M6MA8DN63PD84QDAJZB71D8D","huggingface-tokenizers",null,"Provides the HuggingFace Tokenizers library (Rust core with Python and Node.js bindings) for training and using BPE, WordPiece, and Unigram tokenizers. Covers the pipeline of normalizers, pre-tokenizers, models, post-processors and decoders, padding and truncation, batch encoding, alignment tracking, and conversion to transformers PreTrainedTokenizerFast. Use when training a custom tokenizer or vocabulary on a new corpus, tokenizing large text corpora quickly, mapping tokens back to character offsets for NER or question answering, or configuring normalization and special tokens. Use when wrapping a custom tokenizer for transformers. For SentencePiece models or tiktoken, use those tools instead; for loading a pretrained tokenizer only, AutoTokenizer is enough.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1559,"MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhuggingface-tokenizers\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40],"HuggingFace","huggingface tokenizers","BPE","new","when","tiktoken","ner","loading","Alignment Tracking","Custom Tokenizer","Fast Tokenization","Unigram","WordPiece","rust","Tokenization","back","alignment","conversion","only","question","pipeline","training","2026-11-10 10:12:30",false,"---\nname: huggingface-tokenizers\ndescription: Provides the HuggingFace Tokenizers library (Rust core with Python and Node.js bindings) for training and using BPE, WordPiece, and Unigram tokenizers. Covers the pipeline of normalizers, pre-tokenizers, models, post-processors and decoders, padding and truncation, batch encoding, alignment tracking, and conversion to transformers PreTrainedTokenizerFast. Use when training a custom tokenizer or vocabulary on a new corpus, tokenizing large text corpora quickly, mapping tokens back to character offsets for NER or question answering, or configuring normalization and special tokens. Use when wrapping a custom tokenizer for transformers. For SentencePiece models or tiktoken, use those tools instead; for loading a pretrained tokenizer only, AutoTokenizer is enough.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-training\n  maintainer: Kalaris Labs\n  tags: Tokenization, HuggingFace, BPE, WordPiece, Unigram, Fast Tokenization, Rust, Custom Tokenizer, Alignment Tracking, Production\n  dependencies: tokenizers, transformers, datasets\n---\n\n# HuggingFace Tokenizers - Fast Tokenization for NLP\n\nFast, production-ready tokenizers with Rust performance and Python ease-of-use.\n\n## When to use HuggingFace Tokenizers\n\n**Use HuggingFace Tokenizers when:**\n- Need extremely fast tokenization (\u003C20s per GB of text)\n- Training custom tokenizers from scratch\n- Want alignment tracking (token → original text position)\n- Building production NLP pipelines\n- Need to tokenize large corpora efficiently\n\n**Performance**:\n- **Speed**: \u003C20 seconds to tokenize 1GB on CPU\n- **Implementation**: Rust core with Python\u002FNode.js bindings\n- **Efficiency**: 10-100× faster than pure Python implementations\n\n**Use alternatives instead**:\n- **SentencePiece**: Language-independent, used by T5\u002FALBERT\n- **tiktoken**: OpenAI's BPE tokenizer for GPT models\n- **transformers AutoTokenizer**: Loading pretrained only (uses this library internally)\n\n## Quick start\n\n### Installation\n\n```bash\n# Install tokenizers\npip install tokenizers\n\n# With transformers integration\npip install tokenizers transformers\n```\n\n### Load pretrained tokenizer\n\n```python\nfrom tokenizers import Tokenizer\n\n# Load from HuggingFace Hub\ntokenizer = Tokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Encode text\noutput = tokenizer.encode(\"Hello, how are you?\")\nprint(output.tokens)  # ['hello', ',', 'how', 'are', 'you', '?']\nprint(output.ids)     # [7592, 1010, 2129, 2024, 2017, 1029]\n\n# Decode back\ntext = tokenizer.decode(output.ids)\nprint(text)  # \"hello, how are you?\"\n```\n\n### Train custom BPE tokenizer\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\n\n# Initialize tokenizer with BPE model\ntokenizer = Tokenizer(BPE(unk_token=\"[UNK]\"))\ntokenizer.pre_tokenizer = Whitespace()\n\n# Configure trainer\ntrainer = BpeTrainer(\n    vocab_size=30000,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    min_frequency=2\n)\n\n# Train on files\nfiles = [\"train.txt\", \"validation.txt\"]\ntokenizer.train(files, trainer)\n\n# Save\ntokenizer.save(\"my-tokenizer.json\")\n```\n\n**Training time**: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB\n\n### Batch encoding with padding\n\n```python\n# Enable padding\ntokenizer.enable_padding(pad_id=3, pad_token=\"[PAD]\")\n\n# Encode batch\ntexts = [\"Hello world\", \"This is a longer sentence\"]\nencodings = tokenizer.encode_batch(texts)\n\nfor encoding in encodings:\n    print(encoding.ids)\n# [101, 7592, 2088, 102, 3, 3, 3]\n# [101, 2023, 2003, 1037, 2936, 6251, 102]\n```\n\n## Tokenization algorithms\n\nDetails, code examples and parameter tables: [references\u002Ftokenization-algorithms.md](references\u002Ftokenization-algorithms.md). Read it when this step applies.\n\n## Tokenization pipeline\n\nComplete pipeline: **Normalization → Pre-tokenization → Model → Post-processing**\n\n### Normalization\n\nClean and standardize text:\n\n```python\nfrom tokenizers.normalizers import NFD, StripAccents, Lowercase, Sequence\n\ntokenizer.normalizer = Sequence([\n    NFD(),           # Unicode normalization (decompose)\n    Lowercase(),     # Convert to lowercase\n    StripAccents()   # Remove accents\n])\n\n# Input: \"Héllo WORLD\"\n# After normalization: \"hello world\"\n```\n\n**Common normalizers**:\n- `NFD`, `NFC`, `NFKD`, `NFKC` - Unicode normalization forms\n- `Lowercase()` - Convert to lowercase\n- `StripAccents()` - Remove accents (é → e)\n- `Strip()` - Remove whitespace\n- `Replace(pattern, content)` - Regex replacement\n\n### Pre-tokenization\n\nSplit text into word-like units:\n\n```python\nfrom tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence, ByteLevel\n\n# Split on whitespace and punctuation\ntokenizer.pre_tokenizer = Sequence([\n    Whitespace(),\n    Punctuation()\n])\n\n# Input: \"Hello, world!\"\n# After pre-tokenization: [\"Hello\", \",\", \"world\", \"!\"]\n```\n\n**Common pre-tokenizers**:\n- `Whitespace()` - Split on spaces, tabs, newlines\n- `ByteLevel()` - GPT-2 style byte-level splitting\n- `Punctuation()` - Isolate punctuation\n- `Digits(individual_digits=True)` - Split digits individually\n- `Metaspace()` - Replace spaces with ▁ (SentencePiece style)\n\n### Post-processing\n\nAdd special tokens for model input:\n\n```python\nfrom tokenizers.processors import TemplateProcessing\n\n# BERT-style: [CLS] sentence [SEP]\ntokenizer.post_processor = TemplateProcessing(\n    single=\"[CLS] $A [SEP]\",\n    pair=\"[CLS] $A [SEP] $B [SEP]\",\n    special_tokens=[\n        (\"[CLS]\", 1),\n        (\"[SEP]\", 2),\n    ],\n)\n```\n\n**Common patterns**:\n```python\n# GPT-2: sentence \u003C|endoftext|>\nTemplateProcessing(\n    single=\"$A \u003C|endoftext|>\",\n    special_tokens=[(\"\u003C|endoftext|>\", 50256)]\n)\n\n# RoBERTa: \u003Cs> sentence \u003C\u002Fs>\nTemplateProcessing(\n    single=\"\u003Cs> $A \u003C\u002Fs>\",\n    pair=\"\u003Cs> $A \u003C\u002Fs> \u003C\u002Fs> $B \u003C\u002Fs>\",\n    special_tokens=[(\"\u003Cs>\", 0), (\"\u003C\u002Fs>\", 2)]\n)\n```\n\n## Alignment tracking\n\nTrack token positions in original text:\n\n```python\noutput = tokenizer.encode(\"Hello, world!\")\n\n# Get token offsets\nfor token, offset in zip(output.tokens, output.offsets):\n    start, end = offset\n    print(f\"{token:10} → [{start:2}, {end:2}): {text[start:end]!r}\")\n\n# Output:\n# hello      → [ 0,  5): 'Hello'\n# ,          → [ 5,  6): ','\n# world      → [ 7, 12): 'world'\n# !          → [12, 13): '!'\n```\n\n**Use cases**:\n- Named entity recognition (map predictions back to text)\n- Question answering (extract answer spans)\n- Token classification (align labels to original positions)\n\n## Integration with transformers\n\n### Load with AutoTokenizer\n\n```python\nfrom transformers import AutoTokenizer\n\n# AutoTokenizer automatically uses fast tokenizers\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Check if using fast tokenizer\nprint(tokenizer.is_fast)  # True\n\n# Access underlying tokenizers.Tokenizer\nfast_tokenizer = tokenizer.backend_tokenizer\nprint(type(fast_tokenizer))  # \u003Cclass 'tokenizers.Tokenizer'>\n```\n\n### Convert custom tokenizer to transformers\n\n```python\nfrom tokenizers import Tokenizer\nfrom transformers import PreTrainedTokenizerFast\n\n# Train custom tokenizer\ntokenizer = Tokenizer(BPE())\n# ... train tokenizer ...\ntokenizer.save(\"my-tokenizer.json\")\n\n# Wrap for transformers\ntransformers_tokenizer = PreTrainedTokenizerFast(\n    tokenizer_file=\"my-tokenizer.json\",\n    unk_token=\"[UNK]\",\n    pad_token=\"[PAD]\",\n    cls_token=\"[CLS]\",\n    sep_token=\"[SEP]\",\n    mask_token=\"[MASK]\"\n)\n\n# Use like any transformers tokenizer\noutputs = transformers_tokenizer(\n    \"Hello world\",\n    padding=True,\n    truncation=True,\n    max_length=512,\n    return_tensors=\"pt\"\n)\n```\n\n## Common patterns\n\n### Train from iterator (large datasets)\n\n```python\nfrom datasets import load_dataset\n\n# Load dataset\ndataset = load_dataset(\"wikitext\", \"wikitext-103-raw-v1\", split=\"train\")\n\n# Create batch iterator\ndef batch_iterator(batch_size=1000):\n    for i in range(0, len(dataset), batch_size):\n        yield dataset[i:i + batch_size][\"text\"]\n\n# Train tokenizer\ntokenizer.train_from_iterator(\n    batch_iterator(),\n    trainer=trainer,\n    length=len(dataset)  # For progress bar\n)\n```\n\n**Performance**: Processes 1GB in ~10-20 minutes\n\n### Enable truncation and padding\n\n```python\n# Enable truncation\ntokenizer.enable_truncation(max_length=512)\n\n# Enable padding\ntokenizer.enable_padding(\n    pad_id=tokenizer.token_to_id(\"[PAD]\"),\n    pad_token=\"[PAD]\",\n    length=512  # Fixed length, or None for batch max\n)\n\n# Encode with both\noutput = tokenizer.encode(\"This is a long sentence that will be truncated...\")\nprint(len(output.ids))  # 512\n```\n\n### Multi-processing\n\n```python\nfrom tokenizers import Tokenizer\nfrom multiprocessing import Pool\n\n# Load tokenizer\ntokenizer = Tokenizer.from_file(\"tokenizer.json\")\n\ndef encode_batch(texts):\n    return tokenizer.encode_batch(texts)\n\n# Process large corpus in parallel\nwith Pool(8) as pool:\n    # Split corpus into chunks\n    chunk_size = 1000\n    chunks = [corpus[i:i+chunk_size] for i in range(0, len(corpus), chunk_size)]\n\n    # Encode in parallel\n    results = pool.map(encode_batch, chunks)\n```\n\n**Speedup**: 5-8× with 8 cores\n\n## Performance benchmarks\n\n### Training speed\n\n| Corpus Size | BPE (30k vocab) | WordPiece (30k) | Unigram (8k) |\n|-------------|-----------------|-----------------|--------------|\n| 10 MB       | 15 sec          | 18 sec          | 25 sec       |\n| 100 MB      | 1.5 min         | 2 min           | 4 min        |\n| 1 GB        | 15 min          | 20 min          | 40 min       |\n\n**Hardware**: 16-core CPU, tested on English Wikipedia\n\n### Tokenization speed\n\n| Implementation | 1 GB corpus | Throughput    |\n|----------------|-------------|---------------|\n| Pure Python    | ~20 minutes | ~50 MB\u002Fmin    |\n| HF Tokenizers  | ~15 seconds | ~4 GB\u002Fmin     |\n| **Speedup**    | **80×**     | **80×**       |\n\n**Test**: English text, average sentence length 20 words\n\n### Memory usage\n\n| Task                    | Memory  |\n|-------------------------|---------|\n| Load tokenizer          | ~10 MB  |\n| Train BPE (30k vocab)   | ~200 MB |\n| Encode 1M sentences     | ~500 MB |\n\n## Supported models\n\nPre-trained tokenizers available via `from_pretrained()`:\n\n**BERT family**:\n- `bert-base-uncased`, `bert-large-cased`\n- `distilbert-base-uncased`\n- `roberta-base`, `roberta-large`\n\n**GPT family**:\n- `gpt2`, `gpt2-medium`, `gpt2-large`\n- `distilgpt2`\n\n**T5 family**:\n- `t5-small`, `t5-base`, `t5-large`\n- `google\u002Fflan-t5-xxl`\n\n**Other**:\n- `facebook\u002Fbart-base`, `facebook\u002Fmbart-large-cc25`\n- `albert-base-v2`, `albert-xlarge-v2`\n- `xlm-roberta-base`, `xlm-roberta-large`\n\nBrowse all: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=tokenizers\n\n## References\n\n- **[Training Guide](references\u002Ftraining.md)** - Train custom tokenizers, configure trainers, handle large datasets\n- **[Algorithms Deep Dive](references\u002Falgorithms.md)** - BPE, WordPiece, Unigram explained in detail\n- **[Pipeline Components](references\u002Fpipeline.md)** - Normalizers, pre-tokenizers, post-processors, decoders\n- **[Transformers Integration](references\u002Fintegration.md)** - AutoTokenizer, PreTrainedTokenizerFast, special tokens\n\n## Resources\n\n- **Docs**: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftokenizers\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftokenizers ⭐ 9,000+\n- **Version**: 0.20.0+\n- **Course**: https:\u002F\u002Fhuggingface.co\u002Flearn\u002Fnlp-course\u002Fchapter6\u002F1\n- **Paper**: BPE (Sennrich et al., 2016), WordPiece (Schuster & Nakajima, 2012)\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `transformers`: Hugging Face Transformers for loading Hub models, running pipeline inference, text generation, and Trainer fine-tuning on NLP, vision, audi…\n- `sentencepiece`: Language-independent tokenizer treating text as raw Unicode.\n","# HuggingFace Tokenizers - Fast Tokenization for NLP\n\nFast, production-ready tokenizers with Rust performance and Python ease-of-use.\n\n## When to use HuggingFace Tokenizers\n\n**Use HuggingFace Tokenizers when:**\n- Need extremely fast tokenization (\u003C20s per GB of text)\n- Training custom tokenizers from scratch\n- Want alignment tracking (token → original text position)\n- Building production NLP pipelines\n- Need to tokenize large corpora efficiently\n\n**Performance**:\n- **Speed**: \u003C20 seconds to tokenize 1GB on CPU\n- **Implementation**: Rust core with Python\u002FNode.js bindings\n- **Efficiency**: 10-100× faster than pure Python implementations\n\n**Use alternatives instead**:\n- **SentencePiece**: Language-independent, used by T5\u002FALBERT\n- **tiktoken**: OpenAI's BPE tokenizer for GPT models\n- **transformers AutoTokenizer**: Loading pretrained only (uses this library internally)\n\n## Quick start\n\n### Installation\n\n```bash\n# Install tokenizers\npip install tokenizers\n\n# With transformers integration\npip install tokenizers transformers\n```\n\n### Load pretrained tokenizer\n\n```python\nfrom tokenizers import Tokenizer\n\n# Load from HuggingFace Hub\ntokenizer = Tokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Encode text\noutput = tokenizer.encode(\"Hello, how are you?\")\nprint(output.tokens)  # ['hello', ',', 'how', 'are', 'you', '?']\nprint(output.ids)     # [7592, 1010, 2129, 2024, 2017, 1029]\n\n# Decode back\ntext = tokenizer.decode(output.ids)\nprint(text)  # \"hello, how are you?\"\n```\n\n### Train custom BPE tokenizer\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\n\n# Initialize tokenizer with BPE model\ntokenizer = Tokenizer(BPE(unk_token=\"[UNK]\"))\ntokenizer.pre_tokenizer = Whitespace()\n\n# Configure trainer\ntrainer = BpeTrainer(\n    vocab_size=30000,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    min_frequency=2\n)\n\n# Train on files\nfiles = [\"train.txt\", \"validation.txt\"]\ntokenizer.train(files, trainer)\n\n# Save\ntokenizer.save(\"my-tokenizer.json\")\n```\n\n**Training time**: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB\n\n### Batch encoding with padding\n\n```python\n# Enable padding\ntokenizer.enable_padding(pad_id=3, pad_token=\"[PAD]\")\n\n# Encode batch\ntexts = [\"Hello world\", \"This is a longer sentence\"]\nencodings = tokenizer.encode_batch(texts)\n\nfor encoding in encodings:\n    print(encoding.ids)\n# [101, 7592, 2088, 102, 3, 3, 3]\n# [101, 2023, 2003, 1037, 2936, 6251, 102]\n```\n\n## Tokenization algorithms\n\nDetails, code examples and parameter tables: [references\u002Ftokenization-algorithms.md](references\u002Ftokenization-algorithms.md). Read it when this step applies.\n\n## Tokenization pipeline\n\nComplete pipeline: **Normalization → Pre-tokenization → Model → Post-processing**\n\n### Normalization\n\nClean and standardize text:\n\n```python\nfrom tokenizers.normalizers import NFD, StripAccents, Lowercase, Sequence\n\ntokenizer.normalizer = Sequence([\n    NFD(),           # Unicode normalization (decompose)\n    Lowercase(),     # Convert to lowercase\n    StripAccents()   # Remove accents\n])\n\n# Input: \"Héllo WORLD\"\n# After normalization: \"hello world\"\n```\n\n**Common normalizers**:\n- `NFD`, `NFC`, `NFKD`, `NFKC` - Unicode normalization forms\n- `Lowercase()` - Convert to lowercase\n- `StripAccents()` - Remove accents (é → e)\n- `Strip()` - Remove whitespace\n- `Replace(pattern, content)` - Regex replacement\n\n### Pre-tokenization\n\nSplit text into word-like units:\n\n```python\nfrom tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence, ByteLevel\n\n# Split on whitespace and punctuation\ntokenizer.pre_tokenizer = Sequence([\n    Whitespace(),\n    Punctuation()\n])\n\n# Input: \"Hello, world!\"\n# After pre-tokenization: [\"Hello\", \",\", \"world\", \"!\"]\n```\n\n**Common pre-tokenizers**:\n- `Whitespace()` - Split on spaces, tabs, newlines\n- `ByteLevel()` - GPT-2 style byte-level splitting\n- `Punctuation()` - Isolate punctuation\n- `Digits(individual_digits=True)` - Split digits individually\n- `Metaspace()` - Replace spaces with ▁ (SentencePiece style)\n\n### Post-processing\n\nAdd special tokens for model input:\n\n```python\nfrom tokenizers.processors import TemplateProcessing\n\n# BERT-style: [CLS] sentence [SEP]\ntokenizer.post_processor = TemplateProcessing(\n    single=\"[CLS] $A [SEP]\",\n    pair=\"[CLS] $A [SEP] $B [SEP]\",\n    special_tokens=[\n        (\"[CLS]\", 1),\n        (\"[SEP]\", 2),\n    ],\n)\n```\n\n**Common patterns**:\n```python\n# GPT-2: sentence \u003C|endoftext|>\nTemplateProcessing(\n    single=\"$A \u003C|endoftext|>\",\n    special_tokens=[(\"\u003C|endoftext|>\", 50256)]\n)\n\n# RoBERTa: \u003Cs> sentence \u003C\u002Fs>\nTemplateProcessing(\n    single=\"\u003Cs> $A \u003C\u002Fs>\",\n    pair=\"\u003Cs> $A \u003C\u002Fs> \u003C\u002Fs> $B \u003C\u002Fs>\",\n    special_tokens=[(\"\u003Cs>\", 0), (\"\u003C\u002Fs>\", 2)]\n)\n```\n\n## Alignment tracking\n\nTrack token positions in original text:\n\n```python\noutput = tokenizer.encode(\"Hello, world!\")\n\n# Get token offsets\nfor token, offset in zip(output.tokens, output.offsets):\n    start, end = offset\n    print(f\"{token:10} → [{start:2}, {end:2}): {text[start:end]!r}\")\n\n# Output:\n# hello      → [ 0,  5): 'Hello'\n# ,          → [ 5,  6): ','\n# world      → [ 7, 12): 'world'\n# !          → [12, 13): '!'\n```\n\n**Use cases**:\n- Named entity recognition (map predictions back to text)\n- Question answering (extract answer spans)\n- Token classification (align labels to original positions)\n\n## Integration with transformers\n\n### Load with AutoTokenizer\n\n```python\nfrom transformers import AutoTokenizer\n\n# AutoTokenizer automatically uses fast tokenizers\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Check if using fast tokenizer\nprint(tokenizer.is_fast)  # True\n\n# Access underlying tokenizers.Tokenizer\nfast_tokenizer = tokenizer.backend_tokenizer\nprint(type(fast_tokenizer))  # \u003Cclass 'tokenizers.Tokenizer'>\n```\n\n### Convert custom tokenizer to transformers\n\n```python\nfrom tokenizers import Tokenizer\nfrom transformers import PreTrainedTokenizerFast\n\n# Train custom tokenizer\ntokenizer = Tokenizer(BPE())\n# ... train tokenizer ...\ntokenizer.save(\"my-tokenizer.json\")\n\n# Wrap for transformers\ntransformers_tokenizer = PreTrainedTokenizerFast(\n    tokenizer_file=\"my-tokenizer.json\",\n    unk_token=\"[UNK]\",\n    pad_token=\"[PAD]\",\n    cls_token=\"[CLS]\",\n    sep_token=\"[SEP]\",\n    mask_token=\"[MASK]\"\n)\n\n# Use like any transformers tokenizer\noutputs = transformers_tokenizer(\n    \"Hello world\",\n    padding=True,\n    truncation=True,\n    max_length=512,\n    return_tensors=\"pt\"\n)\n```\n\n## Common patterns\n\n### Train from iterator (large datasets)\n\n```python\nfrom datasets import load_dataset\n\n# Load dataset\ndataset = load_dataset(\"wikitext\", \"wikitext-103-raw-v1\", split=\"train\")\n\n# Create batch iterator\ndef batch_iterator(batch_size=1000):\n    for i in range(0, len(dataset), batch_size):\n        yield dataset[i:i + batch_size][\"text\"]\n\n# Train tokenizer\ntokenizer.train_from_iterator(\n    batch_iterator(),\n    trainer=trainer,\n    length=len(dataset)  # For progress bar\n)\n```\n\n**Performance**: Processes 1GB in ~10-20 minutes\n\n### Enable truncation and padding\n\n```python\n# Enable truncation\ntokenizer.enable_truncation(max_length=512)\n\n# Enable padding\ntokenizer.enable_padding(\n    pad_id=tokenizer.token_to_id(\"[PAD]\"),\n    pad_token=\"[PAD]\",\n    length=512  # Fixed length, or None for batch max\n)\n\n# Encode with both\noutput = tokenizer.encode(\"This is a long sentence that will be truncated...\")\nprint(len(output.ids))  # 512\n```\n\n### Multi-processing\n\n```python\nfrom tokenizers import Tokenizer\nfrom multiprocessing import Pool\n\n# Load tokenizer\ntokenizer = Tokenizer.from_file(\"tokenizer.json\")\n\ndef encode_batch(texts):\n    return tokenizer.encode_batch(texts)\n\n# Process large corpus in parallel\nwith Pool(8) as pool:\n    # Split corpus into chunks\n    chunk_size = 1000\n    chunks = [corpus[i:i+chunk_size] for i in range(0, len(corpus), chunk_size)]\n\n    # Encode in parallel\n    results = pool.map(encode_batch, chunks)\n```\n\n**Speedup**: 5-8× with 8 cores\n\n## Performance benchmarks\n\n### Training speed\n\n| Corpus Size | BPE (30k vocab) | WordPiece (30k) | Unigram (8k) |\n|-------------|-----------------|-----------------|--------------|\n| 10 MB       | 15 sec          | 18 sec          | 25 sec       |\n| 100 MB      | 1.5 min         | 2 min           | 4 min        |\n| 1 GB        | 15 min          | 20 min          | 40 min       |\n\n**Hardware**: 16-core CPU, tested on English Wikipedia\n\n### Tokenization speed\n\n| Implementation | 1 GB corpus | Throughput    |\n|----------------|-------------|---------------|\n| Pure Python    | ~20 minutes | ~50 MB\u002Fmin    |\n| HF Tokenizers  | ~15 seconds | ~4 GB\u002Fmin     |\n| **Speedup**    | **80×**     | **80×**       |\n\n**Test**: English text, average sentence length 20 words\n\n### Memory usage\n\n| Task                    | Memory  |\n|-------------------------|---------|\n| Load tokenizer          | ~10 MB  |\n| Train BPE (30k vocab)   | ~200 MB |\n| Encode 1M sentences     | ~500 MB |\n\n## Supported models\n\nPre-trained tokenizers available via `from_pretrained()`:\n\n**BERT family**:\n- `bert-base-uncased`, `bert-large-cased`\n- `distilbert-base-uncased`\n- `roberta-base`, `roberta-large`\n\n**GPT family**:\n- `gpt2`, `gpt2-medium`, `gpt2-large`\n- `distilgpt2`\n\n**T5 family**:\n- `t5-small`, `t5-base`, `t5-large`\n- `google\u002Fflan-t5-xxl`\n\n**Other**:\n- `facebook\u002Fbart-base`, `facebook\u002Fmbart-large-cc25`\n- `albert-base-v2`, `albert-xlarge-v2`\n- `xlm-roberta-base`, `xlm-roberta-large`\n\nBrowse all: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=tokenizers\n\n## References\n\n- **[Training Guide](references\u002Ftraining.md)** - Train custom tokenizers, configure trainers, handle large datasets\n- **[Algorithms Deep Dive](references\u002Falgorithms.md)** - BPE, WordPiece, Unigram explained in detail\n- **[Pipeline Components](references\u002Fpipeline.md)** - Normalizers, pre-tokenizers, post-processors, decoders\n- **[Transformers Integration](references\u002Fintegration.md)** - AutoTokenizer, PreTrainedTokenizerFast, special tokens\n\n## Resources\n\n- **Docs**: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftokenizers\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftokenizers ⭐ 9,000+\n- **Version**: 0.20.0+\n- **Course**: https:\u002F\u002Fhuggingface.co\u002Flearn\u002Fnlp-course\u002Fchapter6\u002F1\n- **Paper**: BPE (Sennrich et al., 2016), WordPiece (Schuster & Nakajima, 2012)\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `transformers`: Hugging Face Transformers for loading Hub models, running pipeline inference, text generation, and Trainer fine-tuning on NLP, vision, audi…\n- `sentencepiece`: Language-independent tokenizer treating text as raw Unicode.\n",{"name":5,"license":15,"metadata":46,"description":7},{"tags":47,"version":48,"category":49,"maintainer":50,"dependencies":51},"Tokenization, HuggingFace, BPE, WordPiece, Unigram, Fast Tokenization, Rust, Custom Tokenizer, Alignment Tracking, Production","1.0.0","ml-training","Kalaris Labs","tokenizers, transformers, datasets","main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhuggingface-tokenizers",474,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-11 14:40:14",{"source":8,"skillName":5,"commands":63,"prompt":73,"zipAvailable":74},[64,67,70],{"tool":65,"command":66},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill huggingface-tokenizers",{"tool":68,"command":69},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill huggingface-tokenizers",{"tool":71,"command":72},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill huggingface-tokenizers","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhuggingface-tokenizers\u002FSKILL.md",true,[76,82,86,90,94],{"path":77,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Falgorithms.md","REFERENCE","参考文档",15182,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhuggingface-tokenizers\u002Freferences\u002Falgorithms.md",{"path":83,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":84,"sourceHtmlUrl":85},"references\u002Fintegration.md",15389,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhuggingface-tokenizers\u002Freferences\u002Fintegration.md",{"path":87,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":88,"sourceHtmlUrl":89},"references\u002Fpipeline.md",16746,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhuggingface-tokenizers\u002Freferences\u002Fpipeline.md",{"path":91,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":92,"sourceHtmlUrl":93},"references\u002Ftokenization-algorithms.md",2908,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhuggingface-tokenizers\u002Freferences\u002Ftokenization-algorithms.md",{"path":95,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":96,"sourceHtmlUrl":97},"references\u002Ftraining.md",14559,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhuggingface-tokenizers\u002Freferences\u002Ftraining.md",[99,108,115,122,148,173],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":103,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":107},"01M746ZWACVKMZ1BJXJZHYJSSX","migrate-to-e2e","Moves an existing Playwright, Cypress, Selenium, Appium, Detox, Maestro, or TesterArmy suite to e2e one slice at a time, with the old suite running beside the new one until each ported test is proven. Keeps a MIGRATION.md progress file so any later session resumes where the last stopped. Optionally plants a defect to show both suites catch it. Use for \u002Fmigrate-to-e2e, \"migrate our tests to e2e\", \"port these Maestro flows\", \"move our TesterArmy tests\", or when a project has an e2e config and an older suite still running.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Ftester-army\u002Fe2e\u002Fskills\u002Fmigrate-to-e2e\u002FSKILL.md",[],[],"2026-11-11 15:11:14",{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":103,"licenseFileUrl":6,"sourceHtmlUrl":112,"categories":113,"keywords":114,"publishedAt":107},"01M746ZPTAX1Q0FBKAB1CYND7E","e2e","Writes, runs, and debugs end-to-end tests for web and mobile apps with e2e, the agentic test runner, which mixes agent.act goals with exact locator assertions on a Playwright browser or an iOS or Android device. Also drives a project's running app live over the e2e MCP server to look at a screen, check a UI change, take a screenshot or record a video, and try a locator; and runs bug bashes, parallel e2e explore runs with each bug proven by a repro test. Use when a project depends on e2e or has an e2e.config.ts, when asked for end-to-end, browser, mobile, or agentic UI tests, to check a change in the running app, to QA or bug bash an app or branch, or when an e2e run fails.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Ftester-army\u002Fe2e\u002Fskills\u002Fe2e\u002FSKILL.md",[],[],{"skillNo":116,"name":117,"nameZh":6,"description":118,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":103,"licenseFileUrl":6,"sourceHtmlUrl":119,"categories":120,"keywords":121,"publishedAt":107},"01M746ZJN784HVHMDP9MQGCSGG","create-verification-skill","Generate a project-local verify-\u003Capp> skill on top of e2e, so any coding agent can launch the app, check it is worth driving, exercise a feature the way a user does, keep evidence, and bug bash it. Writes the skill, a feature map with one explore charter per feature, and proves it once. Use for \u002Fcreate-verification-skill, \"make a verification skill for this repo\", or when a project has no scripted way to prove UI behavior.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Ftester-army\u002Fe2e\u002Fskills\u002Fcreate-verification-skill\u002FSKILL.md",[],[],{"skillNo":123,"name":124,"nameZh":6,"description":125,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":103,"licenseFileUrl":6,"sourceHtmlUrl":126,"categories":127,"keywords":128,"publishedAt":41},"01M716A6ETNB9B1SQ3GXZC75VR","deja-search","deja-vu memory — search the user's past AI coding sessions with the deja CLI. Holds what they already settled and never wrote into the repo (naming rules, banned or already-tried tools, required checks before merge, preferred values, who owns what). Use before your first edit in a task, before you call a change done or ready to merge, before you recommend a tool, library, design or reviewer, before re-debugging an error that may already be solved, when they say 'didn't we fix this before', 'what did we decide about X' or 'I already have that', and before telling them something on this machine does not exist.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fvshulcz\u002Fdeja-vu\u002Fskills\u002Fdeja-search\u002FSKILL.md",[],[129,130,131,132,133,134,135,136,137,138,139,23,140,141,142,143,144,145,146,147],"search","deja search","never","error","ready","memory","re","decide","fix","design","debugging","you","done","reviewer","io","change","cli","merge","past",{"skillNo":149,"name":150,"nameZh":6,"description":151,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":103,"licenseFileUrl":6,"sourceHtmlUrl":152,"categories":153,"keywords":154,"publishedAt":41},"01M716A3J2W5SENSE9VGMXB375","deja-reflect","End-of-session reflection and memory curation with deja-vu. Review what was decided, what commands broke and what repaired them, and record durable project rules or decisions with deja remember so the next session starts with them.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fvshulcz\u002Fdeja-vu\u002Fskills\u002Fdeja-reflect\u002FSKILL.md",[],[155,156,157,143,158,159,160,161,162,163,164,165,166,167,134,168,169,170,171,172],"deja reflect","reflect","review","with","record","github","next","https","node","Rules","decisions","brew","remember","project rules","project","session","end","reflection",{"skillNo":174,"name":175,"nameZh":6,"description":176,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":103,"licenseFileUrl":6,"sourceHtmlUrl":177,"categories":178,"keywords":190,"publishedAt":41},"01M715T3RR4BAYSR57HPPTJ0PF","revid","Make, edit, export and post AI videos with Revid (revid.ai) through the Revid MCP server. Use whenever the user wants a short-form or long-form video made (from an idea, a script, a URL, an article, a PDF, a tweet, a product page, their own clips, or a song), wants to find what is going viral on TikTok, YouTube, Instagram or X before making something, wants to edit or re-export an existing Revid project, generate a standalone AI image or clip, make a series episode, or publish and schedule a finished video to connected social accounts. Covers setup (hosted MCP with OAuth, or an API key), the 36 tools, credits, polling and the rules for publishing. Routes to the more specific revid-* skills when one fits.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Ftmakersas\u002Frevid-skills\u002Fskills\u002Frevid\u002FSKILL.md",[179,184,187],{"categoryNo":180,"categoryName":181,"taxonomyType":182,"taxonomyTypeDesc":183},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",{"categoryNo":185,"categoryName":186,"taxonomyType":182,"taxonomyTypeDesc":183},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":188,"categoryName":189,"taxonomyType":182,"taxonomyTypeDesc":183},"01JBSEEDCAT000000000000010","后端与接口",[175,191,192,193,194,195,196,169,197,135,198,199,200,201,23,202,203,204,205,206],"URL","Video","short-form","standalone","PDF","viral","tiktok","short","social","MCP","song","image","mcp server","schedule","api","publish"]