[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6ADZGSATBNJ9KV4QF7M1DCH":3,"skill-same:01M6ADZGSATBNJ9KV4QF7M1DCH":149},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":47,"linkOnly":48,"markdownContent":49,"markdownBody":50,"frontMatter":51,"refName":61,"commitSha":62,"skillDirectory":63,"forksCount":64,"repositoryHtmlUrl":65,"repositoryHomepageUrl":66,"ownerHtmlUrl":67,"ownerBlogUrl":68,"ownerDescription":69,"updatedAt":70,"install":71,"resources":84},"01M6ADZGSATBNJ9KV4QF7M1DCH","tokenization-huggingface-tokenizers",null,"Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in \u003C20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding\u002Ftruncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.","davila7\u002Fclaude-code-templates","davila7","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F6216945?v=4","01M6ADEWWFPS191NFM0Y1EX1JH","Daniel Avila","01M6ADJHGM78PHQRD3XZTFDVKK",33286,"MIT","https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Ftokenization-huggingface-tokenizers\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000021","资料检索与研究","SCENARIO","使用场景",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46],"HuggingFace","tokenization huggingface tokenizers","huggingface tokenizers","Tokenization","algorithms","when","track","you","Alignment Tracking","Custom Tokenizer","Fast Tokenization","Unigram","WordPiece","BPE","high-performance","performance","alignment","Production","Transformers","fast","training","research","rust","2026-11-10 10:12:30",false,"---\nname: tokenization-huggingface-tokenizers\ndescription: Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in \u003C20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding\u002Ftruncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  author: Orchestra Research\n  tags: [Tokenization, HuggingFace, BPE, WordPiece, Unigram, Fast Tokenization, Rust, Custom Tokenizer, Alignment Tracking, Production]\n  dependencies: [tokenizers, transformers, datasets]\n---\n\n# HuggingFace Tokenizers - Fast Tokenization for NLP\n\nFast, production-ready tokenizers with Rust performance and Python ease-of-use.\n\n## When to use HuggingFace Tokenizers\n\n**Use HuggingFace Tokenizers when:**\n- Need extremely fast tokenization (\u003C20s per GB of text)\n- Training custom tokenizers from scratch\n- Want alignment tracking (token → original text position)\n- Building production NLP pipelines\n- Need to tokenize large corpora efficiently\n\n**Performance**:\n- **Speed**: \u003C20 seconds to tokenize 1GB on CPU\n- **Implementation**: Rust core with Python\u002FNode.js bindings\n- **Efficiency**: 10-100× faster than pure Python implementations\n\n**Use alternatives instead**:\n- **SentencePiece**: Language-independent, used by T5\u002FALBERT\n- **tiktoken**: OpenAI's BPE tokenizer for GPT models\n- **transformers AutoTokenizer**: Loading pretrained only (uses this library internally)\n\n## Quick start\n\n### Installation\n\n```bash\n# Install tokenizers\npip install tokenizers\n\n# With transformers integration\npip install tokenizers transformers\n```\n\n### Load pretrained tokenizer\n\n```python\nfrom tokenizers import Tokenizer\n\n# Load from HuggingFace Hub\ntokenizer = Tokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Encode text\noutput = tokenizer.encode(\"Hello, how are you?\")\nprint(output.tokens)  # ['hello', ',', 'how', 'are', 'you', '?']\nprint(output.ids)     # [7592, 1010, 2129, 2024, 2017, 1029]\n\n# Decode back\ntext = tokenizer.decode(output.ids)\nprint(text)  # \"hello, how are you?\"\n```\n\n### Train custom BPE tokenizer\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\n\n# Initialize tokenizer with BPE model\ntokenizer = Tokenizer(BPE(unk_token=\"[UNK]\"))\ntokenizer.pre_tokenizer = Whitespace()\n\n# Configure trainer\ntrainer = BpeTrainer(\n    vocab_size=30000,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    min_frequency=2\n)\n\n# Train on files\nfiles = [\"train.txt\", \"validation.txt\"]\ntokenizer.train(files, trainer)\n\n# Save\ntokenizer.save(\"my-tokenizer.json\")\n```\n\n**Training time**: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB\n\n### Batch encoding with padding\n\n```python\n# Enable padding\ntokenizer.enable_padding(pad_id=3, pad_token=\"[PAD]\")\n\n# Encode batch\ntexts = [\"Hello world\", \"This is a longer sentence\"]\nencodings = tokenizer.encode_batch(texts)\n\nfor encoding in encodings:\n    print(encoding.ids)\n# [101, 7592, 2088, 102, 3, 3, 3]\n# [101, 2023, 2003, 1037, 2936, 6251, 102]\n```\n\n## Tokenization algorithms\n\n### BPE (Byte-Pair Encoding)\n\n**How it works**:\n1. Start with character-level vocabulary\n2. Find most frequent character pair\n3. Merge into new token, add to vocabulary\n4. Repeat until vocabulary size reached\n\n**Used by**: GPT-2, GPT-3, RoBERTa, BART, DeBERTa\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import ByteLevel\n\ntokenizer = Tokenizer(BPE(unk_token=\"\u003C|endoftext|>\"))\ntokenizer.pre_tokenizer = ByteLevel()\n\ntrainer = BpeTrainer(\n    vocab_size=50257,\n    special_tokens=[\"\u003C|endoftext|>\"],\n    min_frequency=2\n)\n\ntokenizer.train(files=[\"data.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Handles OOV words well (breaks into subwords)\n- Flexible vocabulary size\n- Good for morphologically rich languages\n\n**Trade-offs**:\n- Tokenization depends on merge order\n- May split common words unexpectedly\n\n### WordPiece\n\n**How it works**:\n1. Start with character vocabulary\n2. Score merge pairs: `frequency(pair) \u002F (frequency(first) × frequency(second))`\n3. Merge highest scoring pair\n4. Repeat until vocabulary size reached\n\n**Used by**: BERT, DistilBERT, MobileBERT\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import WordPiece\nfrom tokenizers.trainers import WordPieceTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\nfrom tokenizers.normalizers import BertNormalizer\n\ntokenizer = Tokenizer(WordPiece(unk_token=\"[UNK]\"))\ntokenizer.normalizer = BertNormalizer(lowercase=True)\ntokenizer.pre_tokenizer = Whitespace()\n\ntrainer = WordPieceTrainer(\n    vocab_size=30522,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    continuing_subword_prefix=\"##\"\n)\n\ntokenizer.train(files=[\"corpus.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Prioritizes meaningful merges (high score = semantically related)\n- Used successfully in BERT (state-of-the-art results)\n\n**Trade-offs**:\n- Unknown words become `[UNK]` if no subword match\n- Saves vocabulary, not merge rules (larger files)\n\n### Unigram\n\n**How it works**:\n1. Start with large vocabulary (all substrings)\n2. Compute loss for corpus with current vocabulary\n3. Remove tokens with minimal impact on loss\n4. Repeat until vocabulary size reached\n\n**Used by**: ALBERT, T5, mBART, XLNet (via SentencePiece)\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import Unigram\nfrom tokenizers.trainers import UnigramTrainer\n\ntokenizer = Tokenizer(Unigram())\n\ntrainer = UnigramTrainer(\n    vocab_size=8000,\n    special_tokens=[\"\u003Cunk>\", \"\u003Cs>\", \"\u003C\u002Fs>\"],\n    unk_token=\"\u003Cunk>\"\n)\n\ntokenizer.train(files=[\"data.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Probabilistic (finds most likely tokenization)\n- Works well for languages without word boundaries\n- Handles diverse linguistic contexts\n\n**Trade-offs**:\n- Computationally expensive to train\n- More hyperparameters to tune\n\n## Tokenization pipeline\n\nComplete pipeline: **Normalization → Pre-tokenization → Model → Post-processing**\n\n### Normalization\n\nClean and standardize text:\n\n```python\nfrom tokenizers.normalizers import NFD, StripAccents, Lowercase, Sequence\n\ntokenizer.normalizer = Sequence([\n    NFD(),           # Unicode normalization (decompose)\n    Lowercase(),     # Convert to lowercase\n    StripAccents()   # Remove accents\n])\n\n# Input: \"Héllo WORLD\"\n# After normalization: \"hello world\"\n```\n\n**Common normalizers**:\n- `NFD`, `NFC`, `NFKD`, `NFKC` - Unicode normalization forms\n- `Lowercase()` - Convert to lowercase\n- `StripAccents()` - Remove accents (é → e)\n- `Strip()` - Remove whitespace\n- `Replace(pattern, content)` - Regex replacement\n\n### Pre-tokenization\n\nSplit text into word-like units:\n\n```python\nfrom tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence, ByteLevel\n\n# Split on whitespace and punctuation\ntokenizer.pre_tokenizer = Sequence([\n    Whitespace(),\n    Punctuation()\n])\n\n# Input: \"Hello, world!\"\n# After pre-tokenization: [\"Hello\", \",\", \"world\", \"!\"]\n```\n\n**Common pre-tokenizers**:\n- `Whitespace()` - Split on spaces, tabs, newlines\n- `ByteLevel()` - GPT-2 style byte-level splitting\n- `Punctuation()` - Isolate punctuation\n- `Digits(individual_digits=True)` - Split digits individually\n- `Metaspace()` - Replace spaces with ▁ (SentencePiece style)\n\n### Post-processing\n\nAdd special tokens for model input:\n\n```python\nfrom tokenizers.processors import TemplateProcessing\n\n# BERT-style: [CLS] sentence [SEP]\ntokenizer.post_processor = TemplateProcessing(\n    single=\"[CLS] $A [SEP]\",\n    pair=\"[CLS] $A [SEP] $B [SEP]\",\n    special_tokens=[\n        (\"[CLS]\", 1),\n        (\"[SEP]\", 2),\n    ],\n)\n```\n\n**Common patterns**:\n```python\n# GPT-2: sentence \u003C|endoftext|>\nTemplateProcessing(\n    single=\"$A \u003C|endoftext|>\",\n    special_tokens=[(\"\u003C|endoftext|>\", 50256)]\n)\n\n# RoBERTa: \u003Cs> sentence \u003C\u002Fs>\nTemplateProcessing(\n    single=\"\u003Cs> $A \u003C\u002Fs>\",\n    pair=\"\u003Cs> $A \u003C\u002Fs> \u003C\u002Fs> $B \u003C\u002Fs>\",\n    special_tokens=[(\"\u003Cs>\", 0), (\"\u003C\u002Fs>\", 2)]\n)\n```\n\n## Alignment tracking\n\nTrack token positions in original text:\n\n```python\noutput = tokenizer.encode(\"Hello, world!\")\n\n# Get token offsets\nfor token, offset in zip(output.tokens, output.offsets):\n    start, end = offset\n    print(f\"{token:10} → [{start:2}, {end:2}): {text[start:end]!r}\")\n\n# Output:\n# hello      → [ 0,  5): 'Hello'\n# ,          → [ 5,  6): ','\n# world      → [ 7, 12): 'world'\n# !          → [12, 13): '!'\n```\n\n**Use cases**:\n- Named entity recognition (map predictions back to text)\n- Question answering (extract answer spans)\n- Token classification (align labels to original positions)\n\n## Integration with transformers\n\n### Load with AutoTokenizer\n\n```python\nfrom transformers import AutoTokenizer\n\n# AutoTokenizer automatically uses fast tokenizers\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Check if using fast tokenizer\nprint(tokenizer.is_fast)  # True\n\n# Access underlying tokenizers.Tokenizer\nfast_tokenizer = tokenizer.backend_tokenizer\nprint(type(fast_tokenizer))  # \u003Cclass 'tokenizers.Tokenizer'>\n```\n\n### Convert custom tokenizer to transformers\n\n```python\nfrom tokenizers import Tokenizer\nfrom transformers import PreTrainedTokenizerFast\n\n# Train custom tokenizer\ntokenizer = Tokenizer(BPE())\n# ... train tokenizer ...\ntokenizer.save(\"my-tokenizer.json\")\n\n# Wrap for transformers\ntransformers_tokenizer = PreTrainedTokenizerFast(\n    tokenizer_file=\"my-tokenizer.json\",\n    unk_token=\"[UNK]\",\n    pad_token=\"[PAD]\",\n    cls_token=\"[CLS]\",\n    sep_token=\"[SEP]\",\n    mask_token=\"[MASK]\"\n)\n\n# Use like any transformers tokenizer\noutputs = transformers_tokenizer(\n    \"Hello world\",\n    padding=True,\n    truncation=True,\n    max_length=512,\n    return_tensors=\"pt\"\n)\n```\n\n## Common patterns\n\n### Train from iterator (large datasets)\n\n```python\nfrom datasets import load_dataset\n\n# Load dataset\ndataset = load_dataset(\"wikitext\", \"wikitext-103-raw-v1\", split=\"train\")\n\n# Create batch iterator\ndef batch_iterator(batch_size=1000):\n    for i in range(0, len(dataset), batch_size):\n        yield dataset[i:i + batch_size][\"text\"]\n\n# Train tokenizer\ntokenizer.train_from_iterator(\n    batch_iterator(),\n    trainer=trainer,\n    length=len(dataset)  # For progress bar\n)\n```\n\n**Performance**: Processes 1GB in ~10-20 minutes\n\n### Enable truncation and padding\n\n```python\n# Enable truncation\ntokenizer.enable_truncation(max_length=512)\n\n# Enable padding\ntokenizer.enable_padding(\n    pad_id=tokenizer.token_to_id(\"[PAD]\"),\n    pad_token=\"[PAD]\",\n    length=512  # Fixed length, or None for batch max\n)\n\n# Encode with both\noutput = tokenizer.encode(\"This is a long sentence that will be truncated...\")\nprint(len(output.ids))  # 512\n```\n\n### Multi-processing\n\n```python\nfrom tokenizers import Tokenizer\nfrom multiprocessing import Pool\n\n# Load tokenizer\ntokenizer = Tokenizer.from_file(\"tokenizer.json\")\n\ndef encode_batch(texts):\n    return tokenizer.encode_batch(texts)\n\n# Process large corpus in parallel\nwith Pool(8) as pool:\n    # Split corpus into chunks\n    chunk_size = 1000\n    chunks = [corpus[i:i+chunk_size] for i in range(0, len(corpus), chunk_size)]\n\n    # Encode in parallel\n    results = pool.map(encode_batch, chunks)\n```\n\n**Speedup**: 5-8× with 8 cores\n\n## Performance benchmarks\n\n### Training speed\n\n| Corpus Size | BPE (30k vocab) | WordPiece (30k) | Unigram (8k) |\n|-------------|-----------------|-----------------|--------------|\n| 10 MB       | 15 sec          | 18 sec          | 25 sec       |\n| 100 MB      | 1.5 min         | 2 min           | 4 min        |\n| 1 GB        | 15 min          | 20 min          | 40 min       |\n\n**Hardware**: 16-core CPU, tested on English Wikipedia\n\n### Tokenization speed\n\n| Implementation | 1 GB corpus | Throughput    |\n|----------------|-------------|---------------|\n| Pure Python    | ~20 minutes | ~50 MB\u002Fmin    |\n| HF Tokenizers  | ~15 seconds | ~4 GB\u002Fmin     |\n| **Speedup**    | **80×**     | **80×**       |\n\n**Test**: English text, average sentence length 20 words\n\n### Memory usage\n\n| Task                    | Memory  |\n|-------------------------|---------|\n| Load tokenizer          | ~10 MB  |\n| Train BPE (30k vocab)   | ~200 MB |\n| Encode 1M sentences     | ~500 MB |\n\n## Supported models\n\nPre-trained tokenizers available via `from_pretrained()`:\n\n**BERT family**:\n- `bert-base-uncased`, `bert-large-cased`\n- `distilbert-base-uncased`\n- `roberta-base`, `roberta-large`\n\n**GPT family**:\n- `gpt2`, `gpt2-medium`, `gpt2-large`\n- `distilgpt2`\n\n**T5 family**:\n- `t5-small`, `t5-base`, `t5-large`\n- `google\u002Fflan-t5-xxl`\n\n**Other**:\n- `facebook\u002Fbart-base`, `facebook\u002Fmbart-large-cc25`\n- `albert-base-v2`, `albert-xlarge-v2`\n- `xlm-roberta-base`, `xlm-roberta-large`\n\nBrowse all: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=tokenizers\n\n## References\n\n- **[Training Guide](references\u002Ftraining.md)** - Train custom tokenizers, configure trainers, handle large datasets\n- **[Algorithms Deep Dive](references\u002Falgorithms.md)** - BPE, WordPiece, Unigram explained in detail\n- **[Pipeline Components](references\u002Fpipeline.md)** - Normalizers, pre-tokenizers, post-processors, decoders\n- **[Transformers Integration](references\u002Fintegration.md)** - AutoTokenizer, PreTrainedTokenizerFast, special tokens\n\n## Resources\n\n- **Docs**: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftokenizers\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftokenizers ⭐ 9,000+\n- **Version**: 0.20.0+\n- **Course**: https:\u002F\u002Fhuggingface.co\u002Flearn\u002Fnlp-course\u002Fchapter6\u002F1\n- **Paper**: BPE (Sennrich et al., 2016), WordPiece (Schuster & Nakajima, 2012)\n\n\n","# HuggingFace Tokenizers - Fast Tokenization for NLP\n\nFast, production-ready tokenizers with Rust performance and Python ease-of-use.\n\n## When to use HuggingFace Tokenizers\n\n**Use HuggingFace Tokenizers when:**\n- Need extremely fast tokenization (\u003C20s per GB of text)\n- Training custom tokenizers from scratch\n- Want alignment tracking (token → original text position)\n- Building production NLP pipelines\n- Need to tokenize large corpora efficiently\n\n**Performance**:\n- **Speed**: \u003C20 seconds to tokenize 1GB on CPU\n- **Implementation**: Rust core with Python\u002FNode.js bindings\n- **Efficiency**: 10-100× faster than pure Python implementations\n\n**Use alternatives instead**:\n- **SentencePiece**: Language-independent, used by T5\u002FALBERT\n- **tiktoken**: OpenAI's BPE tokenizer for GPT models\n- **transformers AutoTokenizer**: Loading pretrained only (uses this library internally)\n\n## Quick start\n\n### Installation\n\n```bash\n# Install tokenizers\npip install tokenizers\n\n# With transformers integration\npip install tokenizers transformers\n```\n\n### Load pretrained tokenizer\n\n```python\nfrom tokenizers import Tokenizer\n\n# Load from HuggingFace Hub\ntokenizer = Tokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Encode text\noutput = tokenizer.encode(\"Hello, how are you?\")\nprint(output.tokens)  # ['hello', ',', 'how', 'are', 'you', '?']\nprint(output.ids)     # [7592, 1010, 2129, 2024, 2017, 1029]\n\n# Decode back\ntext = tokenizer.decode(output.ids)\nprint(text)  # \"hello, how are you?\"\n```\n\n### Train custom BPE tokenizer\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\n\n# Initialize tokenizer with BPE model\ntokenizer = Tokenizer(BPE(unk_token=\"[UNK]\"))\ntokenizer.pre_tokenizer = Whitespace()\n\n# Configure trainer\ntrainer = BpeTrainer(\n    vocab_size=30000,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    min_frequency=2\n)\n\n# Train on files\nfiles = [\"train.txt\", \"validation.txt\"]\ntokenizer.train(files, trainer)\n\n# Save\ntokenizer.save(\"my-tokenizer.json\")\n```\n\n**Training time**: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB\n\n### Batch encoding with padding\n\n```python\n# Enable padding\ntokenizer.enable_padding(pad_id=3, pad_token=\"[PAD]\")\n\n# Encode batch\ntexts = [\"Hello world\", \"This is a longer sentence\"]\nencodings = tokenizer.encode_batch(texts)\n\nfor encoding in encodings:\n    print(encoding.ids)\n# [101, 7592, 2088, 102, 3, 3, 3]\n# [101, 2023, 2003, 1037, 2936, 6251, 102]\n```\n\n## Tokenization algorithms\n\n### BPE (Byte-Pair Encoding)\n\n**How it works**:\n1. Start with character-level vocabulary\n2. Find most frequent character pair\n3. Merge into new token, add to vocabulary\n4. Repeat until vocabulary size reached\n\n**Used by**: GPT-2, GPT-3, RoBERTa, BART, DeBERTa\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import ByteLevel\n\ntokenizer = Tokenizer(BPE(unk_token=\"\u003C|endoftext|>\"))\ntokenizer.pre_tokenizer = ByteLevel()\n\ntrainer = BpeTrainer(\n    vocab_size=50257,\n    special_tokens=[\"\u003C|endoftext|>\"],\n    min_frequency=2\n)\n\ntokenizer.train(files=[\"data.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Handles OOV words well (breaks into subwords)\n- Flexible vocabulary size\n- Good for morphologically rich languages\n\n**Trade-offs**:\n- Tokenization depends on merge order\n- May split common words unexpectedly\n\n### WordPiece\n\n**How it works**:\n1. Start with character vocabulary\n2. Score merge pairs: `frequency(pair) \u002F (frequency(first) × frequency(second))`\n3. Merge highest scoring pair\n4. Repeat until vocabulary size reached\n\n**Used by**: BERT, DistilBERT, MobileBERT\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import WordPiece\nfrom tokenizers.trainers import WordPieceTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\nfrom tokenizers.normalizers import BertNormalizer\n\ntokenizer = Tokenizer(WordPiece(unk_token=\"[UNK]\"))\ntokenizer.normalizer = BertNormalizer(lowercase=True)\ntokenizer.pre_tokenizer = Whitespace()\n\ntrainer = WordPieceTrainer(\n    vocab_size=30522,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    continuing_subword_prefix=\"##\"\n)\n\ntokenizer.train(files=[\"corpus.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Prioritizes meaningful merges (high score = semantically related)\n- Used successfully in BERT (state-of-the-art results)\n\n**Trade-offs**:\n- Unknown words become `[UNK]` if no subword match\n- Saves vocabulary, not merge rules (larger files)\n\n### Unigram\n\n**How it works**:\n1. Start with large vocabulary (all substrings)\n2. Compute loss for corpus with current vocabulary\n3. Remove tokens with minimal impact on loss\n4. Repeat until vocabulary size reached\n\n**Used by**: ALBERT, T5, mBART, XLNet (via SentencePiece)\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import Unigram\nfrom tokenizers.trainers import UnigramTrainer\n\ntokenizer = Tokenizer(Unigram())\n\ntrainer = UnigramTrainer(\n    vocab_size=8000,\n    special_tokens=[\"\u003Cunk>\", \"\u003Cs>\", \"\u003C\u002Fs>\"],\n    unk_token=\"\u003Cunk>\"\n)\n\ntokenizer.train(files=[\"data.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Probabilistic (finds most likely tokenization)\n- Works well for languages without word boundaries\n- Handles diverse linguistic contexts\n\n**Trade-offs**:\n- Computationally expensive to train\n- More hyperparameters to tune\n\n## Tokenization pipeline\n\nComplete pipeline: **Normalization → Pre-tokenization → Model → Post-processing**\n\n### Normalization\n\nClean and standardize text:\n\n```python\nfrom tokenizers.normalizers import NFD, StripAccents, Lowercase, Sequence\n\ntokenizer.normalizer = Sequence([\n    NFD(),           # Unicode normalization (decompose)\n    Lowercase(),     # Convert to lowercase\n    StripAccents()   # Remove accents\n])\n\n# Input: \"Héllo WORLD\"\n# After normalization: \"hello world\"\n```\n\n**Common normalizers**:\n- `NFD`, `NFC`, `NFKD`, `NFKC` - Unicode normalization forms\n- `Lowercase()` - Convert to lowercase\n- `StripAccents()` - Remove accents (é → e)\n- `Strip()` - Remove whitespace\n- `Replace(pattern, content)` - Regex replacement\n\n### Pre-tokenization\n\nSplit text into word-like units:\n\n```python\nfrom tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence, ByteLevel\n\n# Split on whitespace and punctuation\ntokenizer.pre_tokenizer = Sequence([\n    Whitespace(),\n    Punctuation()\n])\n\n# Input: \"Hello, world!\"\n# After pre-tokenization: [\"Hello\", \",\", \"world\", \"!\"]\n```\n\n**Common pre-tokenizers**:\n- `Whitespace()` - Split on spaces, tabs, newlines\n- `ByteLevel()` - GPT-2 style byte-level splitting\n- `Punctuation()` - Isolate punctuation\n- `Digits(individual_digits=True)` - Split digits individually\n- `Metaspace()` - Replace spaces with ▁ (SentencePiece style)\n\n### Post-processing\n\nAdd special tokens for model input:\n\n```python\nfrom tokenizers.processors import TemplateProcessing\n\n# BERT-style: [CLS] sentence [SEP]\ntokenizer.post_processor = TemplateProcessing(\n    single=\"[CLS] $A [SEP]\",\n    pair=\"[CLS] $A [SEP] $B [SEP]\",\n    special_tokens=[\n        (\"[CLS]\", 1),\n        (\"[SEP]\", 2),\n    ],\n)\n```\n\n**Common patterns**:\n```python\n# GPT-2: sentence \u003C|endoftext|>\nTemplateProcessing(\n    single=\"$A \u003C|endoftext|>\",\n    special_tokens=[(\"\u003C|endoftext|>\", 50256)]\n)\n\n# RoBERTa: \u003Cs> sentence \u003C\u002Fs>\nTemplateProcessing(\n    single=\"\u003Cs> $A \u003C\u002Fs>\",\n    pair=\"\u003Cs> $A \u003C\u002Fs> \u003C\u002Fs> $B \u003C\u002Fs>\",\n    special_tokens=[(\"\u003Cs>\", 0), (\"\u003C\u002Fs>\", 2)]\n)\n```\n\n## Alignment tracking\n\nTrack token positions in original text:\n\n```python\noutput = tokenizer.encode(\"Hello, world!\")\n\n# Get token offsets\nfor token, offset in zip(output.tokens, output.offsets):\n    start, end = offset\n    print(f\"{token:10} → [{start:2}, {end:2}): {text[start:end]!r}\")\n\n# Output:\n# hello      → [ 0,  5): 'Hello'\n# ,          → [ 5,  6): ','\n# world      → [ 7, 12): 'world'\n# !          → [12, 13): '!'\n```\n\n**Use cases**:\n- Named entity recognition (map predictions back to text)\n- Question answering (extract answer spans)\n- Token classification (align labels to original positions)\n\n## Integration with transformers\n\n### Load with AutoTokenizer\n\n```python\nfrom transformers import AutoTokenizer\n\n# AutoTokenizer automatically uses fast tokenizers\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Check if using fast tokenizer\nprint(tokenizer.is_fast)  # True\n\n# Access underlying tokenizers.Tokenizer\nfast_tokenizer = tokenizer.backend_tokenizer\nprint(type(fast_tokenizer))  # \u003Cclass 'tokenizers.Tokenizer'>\n```\n\n### Convert custom tokenizer to transformers\n\n```python\nfrom tokenizers import Tokenizer\nfrom transformers import PreTrainedTokenizerFast\n\n# Train custom tokenizer\ntokenizer = Tokenizer(BPE())\n# ... train tokenizer ...\ntokenizer.save(\"my-tokenizer.json\")\n\n# Wrap for transformers\ntransformers_tokenizer = PreTrainedTokenizerFast(\n    tokenizer_file=\"my-tokenizer.json\",\n    unk_token=\"[UNK]\",\n    pad_token=\"[PAD]\",\n    cls_token=\"[CLS]\",\n    sep_token=\"[SEP]\",\n    mask_token=\"[MASK]\"\n)\n\n# Use like any transformers tokenizer\noutputs = transformers_tokenizer(\n    \"Hello world\",\n    padding=True,\n    truncation=True,\n    max_length=512,\n    return_tensors=\"pt\"\n)\n```\n\n## Common patterns\n\n### Train from iterator (large datasets)\n\n```python\nfrom datasets import load_dataset\n\n# Load dataset\ndataset = load_dataset(\"wikitext\", \"wikitext-103-raw-v1\", split=\"train\")\n\n# Create batch iterator\ndef batch_iterator(batch_size=1000):\n    for i in range(0, len(dataset), batch_size):\n        yield dataset[i:i + batch_size][\"text\"]\n\n# Train tokenizer\ntokenizer.train_from_iterator(\n    batch_iterator(),\n    trainer=trainer,\n    length=len(dataset)  # For progress bar\n)\n```\n\n**Performance**: Processes 1GB in ~10-20 minutes\n\n### Enable truncation and padding\n\n```python\n# Enable truncation\ntokenizer.enable_truncation(max_length=512)\n\n# Enable padding\ntokenizer.enable_padding(\n    pad_id=tokenizer.token_to_id(\"[PAD]\"),\n    pad_token=\"[PAD]\",\n    length=512  # Fixed length, or None for batch max\n)\n\n# Encode with both\noutput = tokenizer.encode(\"This is a long sentence that will be truncated...\")\nprint(len(output.ids))  # 512\n```\n\n### Multi-processing\n\n```python\nfrom tokenizers import Tokenizer\nfrom multiprocessing import Pool\n\n# Load tokenizer\ntokenizer = Tokenizer.from_file(\"tokenizer.json\")\n\ndef encode_batch(texts):\n    return tokenizer.encode_batch(texts)\n\n# Process large corpus in parallel\nwith Pool(8) as pool:\n    # Split corpus into chunks\n    chunk_size = 1000\n    chunks = [corpus[i:i+chunk_size] for i in range(0, len(corpus), chunk_size)]\n\n    # Encode in parallel\n    results = pool.map(encode_batch, chunks)\n```\n\n**Speedup**: 5-8× with 8 cores\n\n## Performance benchmarks\n\n### Training speed\n\n| Corpus Size | BPE (30k vocab) | WordPiece (30k) | Unigram (8k) |\n|-------------|-----------------|-----------------|--------------|\n| 10 MB       | 15 sec          | 18 sec          | 25 sec       |\n| 100 MB      | 1.5 min         | 2 min           | 4 min        |\n| 1 GB        | 15 min          | 20 min          | 40 min       |\n\n**Hardware**: 16-core CPU, tested on English Wikipedia\n\n### Tokenization speed\n\n| Implementation | 1 GB corpus | Throughput    |\n|----------------|-------------|---------------|\n| Pure Python    | ~20 minutes | ~50 MB\u002Fmin    |\n| HF Tokenizers  | ~15 seconds | ~4 GB\u002Fmin     |\n| **Speedup**    | **80×**     | **80×**       |\n\n**Test**: English text, average sentence length 20 words\n\n### Memory usage\n\n| Task                    | Memory  |\n|-------------------------|---------|\n| Load tokenizer          | ~10 MB  |\n| Train BPE (30k vocab)   | ~200 MB |\n| Encode 1M sentences     | ~500 MB |\n\n## Supported models\n\nPre-trained tokenizers available via `from_pretrained()`:\n\n**BERT family**:\n- `bert-base-uncased`, `bert-large-cased`\n- `distilbert-base-uncased`\n- `roberta-base`, `roberta-large`\n\n**GPT family**:\n- `gpt2`, `gpt2-medium`, `gpt2-large`\n- `distilgpt2`\n\n**T5 family**:\n- `t5-small`, `t5-base`, `t5-large`\n- `google\u002Fflan-t5-xxl`\n\n**Other**:\n- `facebook\u002Fbart-base`, `facebook\u002Fmbart-large-cc25`\n- `albert-base-v2`, `albert-xlarge-v2`\n- `xlm-roberta-base`, `xlm-roberta-large`\n\nBrowse all: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=tokenizers\n\n## References\n\n- **[Training Guide](references\u002Ftraining.md)** - Train custom tokenizers, configure trainers, handle large datasets\n- **[Algorithms Deep Dive](references\u002Falgorithms.md)** - BPE, WordPiece, Unigram explained in detail\n- **[Pipeline Components](references\u002Fpipeline.md)** - Normalizers, pre-tokenizers, post-processors, decoders\n- **[Transformers Integration](references\u002Fintegration.md)** - AutoTokenizer, PreTrainedTokenizerFast, special tokens\n\n## Resources\n\n- **Docs**: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftokenizers\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftokenizers ⭐ 9,000+\n- **Version**: 0.20.0+\n- **Course**: https:\u002F\u002Fhuggingface.co\u002Flearn\u002Fnlp-course\u002Fchapter6\u002F1\n- **Paper**: BPE (Sennrich et al., 2016), WordPiece (Schuster & Nakajima, 2012)\n\n\n",{"name":5,"license":15,"metadata":52,"description":7},{"tags":53,"author":55,"version":56,"dependencies":57},[27,24,37,36,35,34,54,33,32,41],"Rust","Orchestra Research","1.0.0",[58,59,60],"tokenizers","transformers","datasets","main","2b749c82b5d11fac39d9a6e021ce3f5b8a541bd1","cli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Ftokenization-huggingface-tokenizers",3909,"https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates","https:\u002F\u002Faitmpl.com","https:\u002F\u002Fgithub.com\u002Fdavila7","https:\u002F\u002Fdanielavila.me","Building AI dev tools with LLMs","2026-11-11 11:53:30",{"source":8,"skillName":5,"commands":72,"prompt":82,"zipAvailable":83},[73,76,79],{"tool":74,"command":75},"npx","npx skills add davila7\u002Fclaude-code-templates --skill tokenization-huggingface-tokenizers",{"tool":77,"command":78},"bunx","bunx skills add davila7\u002Fclaude-code-templates --skill tokenization-huggingface-tokenizers",{"tool":80,"command":81},"pnpm","pnpm dlx skills add davila7\u002Fclaude-code-templates --skill tokenization-huggingface-tokenizers","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Ftokenization-huggingface-tokenizers\u002FSKILL.md",true,[85,91,95,101,105,109,113,117,121,125,129,133,137,141,145],{"path":86,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":89,"sourceHtmlUrl":90},"evals\u002Fmodel-expectations.md","REFERENCE","参考文档",5757,"https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fenterprise-communication\u002Fsession-handoff\u002Fevals\u002Fmodel-expectations.md",{"path":92,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":93,"sourceHtmlUrl":94},"evals\u002Fresults-opus-baseline.md",2704,"https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fenterprise-communication\u002Fsession-handoff\u002Fevals\u002Fresults-opus-baseline.md",{"path":96,"resourceType":97,"resourceTypeDesc":98,"sizeBytes":99,"sourceHtmlUrl":100},"evals\u002Fsetup_test_env.py","SCRIPT","脚本",10293,"https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fenterprise-communication\u002Fsession-handoff\u002Fevals\u002Fsetup_test_env.py",{"path":102,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":103,"sourceHtmlUrl":104},"evals\u002Ftest-scenarios.md",5306,"https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fenterprise-communication\u002Fsession-handoff\u002Fevals\u002Ftest-scenarios.md",{"path":106,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":107,"sourceHtmlUrl":108},"README.md",7033,"https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fenterprise-communication\u002Fsession-handoff\u002FREADME.md",{"path":110,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":111,"sourceHtmlUrl":112},"references\u002Falgorithms.md",15182,"https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Ftokenization-huggingface-tokenizers\u002Freferences\u002Falgorithms.md",{"path":114,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":115,"sourceHtmlUrl":116},"references\u002Fhandoff-template.md",3963,"https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fenterprise-communication\u002Fsession-handoff\u002Freferences\u002Fhandoff-template.md",{"path":118,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":119,"sourceHtmlUrl":120},"references\u002Fintegration.md",15389,"https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Ftokenization-huggingface-tokenizers\u002Freferences\u002Fintegration.md",{"path":122,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":123,"sourceHtmlUrl":124},"references\u002Fpipeline.md",16746,"https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Ftokenization-huggingface-tokenizers\u002Freferences\u002Fpipeline.md",{"path":126,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":127,"sourceHtmlUrl":128},"references\u002Fresume-checklist.md",2782,"https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fenterprise-communication\u002Fsession-handoff\u002Freferences\u002Fresume-checklist.md",{"path":130,"resourceType":87,"resourceTypeDesc":88,"sizeBytes":131,"sourceHtmlUrl":132},"references\u002Ftraining.md",14559,"https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Ftokenization-huggingface-tokenizers\u002Freferences\u002Ftraining.md",{"path":134,"resourceType":97,"resourceTypeDesc":98,"sizeBytes":135,"sourceHtmlUrl":136},"scripts\u002Fcheck_staleness.py",13721,"https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fenterprise-communication\u002Fsession-handoff\u002Fscripts\u002Fcheck_staleness.py",{"path":138,"resourceType":97,"resourceTypeDesc":98,"sizeBytes":139,"sourceHtmlUrl":140},"scripts\u002Fcreate_handoff.py",10666,"https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fenterprise-communication\u002Fsession-handoff\u002Fscripts\u002Fcreate_handoff.py",{"path":142,"resourceType":97,"resourceTypeDesc":98,"sizeBytes":143,"sourceHtmlUrl":144},"scripts\u002Flist_handoffs.py",3813,"https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fenterprise-communication\u002Fsession-handoff\u002Fscripts\u002Flist_handoffs.py",{"path":146,"resourceType":97,"resourceTypeDesc":98,"sizeBytes":147,"sourceHtmlUrl":148},"scripts\u002Fvalidate_handoff.py",10821,"https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fenterprise-communication\u002Fsession-handoff\u002Fscripts\u002Fvalidate_handoff.py",[150,178,205,238,273,297],{"skillNo":151,"name":152,"nameZh":6,"description":153,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":154,"categories":155,"keywords":159,"publishedAt":47},"01M70NHBTQ8BV0A2SCAZXEPHCX","wealthville-pool-scoring","Score DeFi liquidity pools before providing liquidity. Returns a 0-100 pool score and an Enter\u002FHold\u002FExit\u002FReduce\u002FAvoid verdict with calibrated confidence for ~68,800 Solana pools and 575 EVM pools, plus a public miss-inclusive track record. Use when asked whether a pool is worth LPing into, which pools look best right now, or how accurate the scores have actually been.","https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fweb-data\u002Fwealthville-pool-scoring\u002FSKILL.md",[156],{"categoryNo":157,"categoryName":158,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发",[160,161,162,163,164,165,166,167,168,29,169,170,171,30,172,173,174,175,176,177],"scoring","pool","wealthville pool scoring","liquidity","which","for","exit","now","use","reduce","with","record","verdict","evm","solana","how","public","look",{"skillNo":179,"name":180,"nameZh":6,"description":181,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":182,"categories":183,"keywords":185,"publishedAt":47},"01M70MB6VR8ERJ4EM8D5KHBMEV","technical-debt-auditor","Scans a codebase and produces a quantified technical debt report with a prioritized remediation plan. Analyzes dead code, TODO\u002FFIXME\u002FHACK markers, dependency staleness, complexity hotspots, code duplication signals, test coverage gaps, unused exports, and outdated patterns. Use this skill when the user asks to audit their codebase for tech debt, wants a health check of their code quality, says things like \"where should I focus refactoring\", \"find code smells\", \"audit my codebase\", \"what technical debt do we have\", \"prioritize cleanup work\", or asks about code quality metrics. Also trigger when the user mentions \"code rot\", \"legacy cleanup\", \"refactoring priorities\", or wants to understand the overall health of a project before a major effort. Trigger even if the user doesn't say \"technical debt\" explicitly — any request to systematically assess code quality and identify improvement priorities belongs here.","https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fdevelopment\u002Ftechnical-debt-auditor\u002FSKILL.md",[184],{"categoryNo":157,"categoryName":158,"taxonomyType":21,"taxonomyTypeDesc":22},[186,187,188,189,190,191,192,193,194,195,196,29,197,198,199,200,201,202,203,204],"technical debt","debt","technical","technical debt auditor","auditor","coverage","complexity","markers","project","cleanup","unused","code","todo","work","plan","focus","signals","audit","dependency",{"skillNo":206,"name":207,"nameZh":6,"description":208,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":209,"categories":210,"keywords":220,"publishedAt":47},"01M70KPYH9QNHCNME5GJQGE22K","readonly-sql-guard","Make an agent's database access provably read-only. Use when an agent runs caller-supplied SQL against a production\u002FERP\u002Fanalytics database (a \"query\" MCP tool, text-to-SQL, an admin console), or when reviewing DB-access code for write\u002FDDL\u002Fside-effect\u002Ftransaction-escape risks. Checks what a statement CALLS, not just its shape.","https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fdatabase\u002Freadonly-sql-guard\u002FSKILL.md",[211,214,215],{"categoryNo":212,"categoryName":213,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000010","后端与接口",{"categoryNo":157,"categoryName":158,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":216,"categoryName":217,"taxonomyType":218,"taxonomyTypeDesc":219},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",[221,222,223,224,225,226,227,165,228,229,230,231,232,233,234,168,235,236,29,237],"readonly","guard","sql guard","sql","readonly sql guard","erp","effect","read-only","db","just","tool","calls","only","MCP","access","database","query",{"skillNo":239,"name":240,"nameZh":6,"description":241,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":242,"licenseFileUrl":243,"sourceHtmlUrl":244,"categories":245,"keywords":249,"publishedAt":47},"01M6AFY6F7CC2BBZP1QD6VTNVC","web-artifacts-builder","Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn\u002Fui). Use for complex artifacts requiring state management, routing, or shadcn\u002Fui components - not for simple single-file HTML\u002FJSX artifacts.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Futilities\u002Fweb-artifacts-builder\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Futilities\u002Fweb-artifacts-builder\u002FSKILL.md",[246],{"categoryNo":247,"categoryName":248,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000009","前端开发",[250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,165,272],"artifacts builder","web","web artifacts builder","artifacts","builder","state","routing","state management","component","css","frontend","ui","Tools","file","components","shadcn","react","Tailwind","suite","jsx","html","single-file","ui components",{"skillNo":274,"name":275,"nameZh":6,"description":276,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":242,"licenseFileUrl":277,"sourceHtmlUrl":278,"categories":279,"keywords":284,"publishedAt":47},"01M6AFWF33SFV9Z3SF4PEEE4BF","webapp-testing","Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.","https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fsecurity\u002Fwebapp-testing\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fsecurity\u002Fwebapp-testing\u002FSKILL.md",[280,283],{"categoryNo":281,"categoryName":282,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000008","测试与质量",{"categoryNo":247,"categoryName":248,"taxonomyType":21,"taxonomyTypeDesc":22},[285,286,287,288,289,251,290,260,261,291,292,293,294,165,295,170,296],"webapp testing","testing","webapp","browser","playwright","applications","debugging","toolkit","logs","screenshots","local","behavior",{"skillNo":298,"name":299,"nameZh":6,"description":300,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":242,"licenseFileUrl":301,"sourceHtmlUrl":302,"categories":303,"keywords":311,"publishedAt":47},"01M6AF67Q67JJJ5V51JR2P52KA","discernment-nudge","After you give a substantive answer or draft that the user may act on — advice or recommendations, drafted artifacts such as goals, plans, pitches, proposals, or emails, estimates or projections, analysis or interpretation of data, factual claims they may rely on, or a multi-step argument — invoke this skill BEFORE finalizing your reply and then, if it applies, append 2-3 short follow-up questions, each tied to something specific in what you just produced, that help the user check key facts, probe the reasoning or assumptions, and notice missing context. Do this at most once per conversation. Skip it when the user asked a trivial how-to or simple lookup, wants a purely educational explanation, asked you only to format, convert, or assemble a file from content they provided, is writing code they will run, is doing creative writing or casual chat, or already asked you to double-check, cite, or review — the skill file explains these boundaries and the exact output format.","https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fproductivity\u002Fdiscernment-nudge\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fdavila7\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fproductivity\u002Fdiscernment-nudge\u002FSKILL.md",[304,307,308],{"categoryNo":305,"categoryName":306,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":157,"categoryName":158,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":309,"categoryName":310,"taxonomyType":218,"taxonomyTypeDesc":219},"01JBSEEDCAT000000000000036","内容创作者",[312,313,314,315,316,317,318,319,253,320,263,197,321,322,323,324,325,233,326,29,31,327,328,329,330,175,331],"discernment nudge","nudge","interpretation","help","run","reasoning","data","writing","lookup","content","review","conversation","short","up","invoke","analysis","claims","argument","assemble","recommendations"]