[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M758WTKNTVZJCNH9BYGDTY5W":3,"skill-same:01M758WTKNTVZJCNH9BYGDTY5W":75},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":39,"commitSha":40,"skillDirectory":41,"forksCount":14,"repositoryHtmlUrl":42,"repositoryHomepageUrl":6,"ownerHtmlUrl":43,"ownerBlogUrl":44,"ownerDescription":45,"updatedAt":46,"install":47,"resources":60},"01M758WTKNTVZJCNH9BYGDTY5W","sentencepiece",null,"Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences\u002Fsec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.","zengrong233\u002Fdevin-skills","zengrong233","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F55529920?v=4","01M758DNYKQQCQX87XFVAGX5MW","songzengrong","01M758DNZVEJJQNM71DSRVHFEB",0,"MIT","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fsentencepiece\u002FSKILL.md",[],[],"2026-11-12 13:55:43",false,"---\nname: sentencepiece\ndescription: Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences\u002Fsec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Tokenization, SentencePiece, Language-Independent, BPE, Unigram, Multilingual, CJK Languages, Unicode, Deterministic, Google]\ndependencies: [sentencepiece, transformers]\n---\n\n# SentencePiece - Language-Independent Tokenization\n\nUnsupervised tokenizer that works on raw text without language-specific preprocessing.\n\n## When to use SentencePiece\n\n**Use SentencePiece when:**\n- Building multilingual models (no language-specific rules)\n- Working with CJK languages (Chinese, Japanese, Korean)\n- Need reproducible tokenization (deterministic vocabulary)\n- Want to train on raw text (no pre-tokenization needed)\n- Require lightweight deployment (6MB memory, 50k sentences\u002Fsec)\n\n**Performance**:\n- **Speed**: 50,000 sentences\u002Fsec\n- **Memory**: ~6MB for loaded model\n- **Languages**: All (language-independent)\n\n**Use alternatives instead**:\n- **HuggingFace Tokenizers**: Faster training, more flexibility\n- **tiktoken**: OpenAI models (GPT-3.5\u002F4)\n- **BERT WordPiece**: English-centric tasks\n\n## Quick start\n\n### Installation\n\n```bash\n# Python\npip install sentencepiece\n\n# C++ (requires CMake)\ngit clone https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fsentencepiece.git\ncd sentencepiece\nmkdir build && cd build\ncmake .. && make -j $(nproc)\nsudo make install\n```\n\n### Train model\n\n```bash\n# Command-line (BPE with 8000 vocab)\nspm_train --input=data.txt --model_prefix=m --vocab_size=8000 --model_type=bpe\n\n# Python API\nimport sentencepiece as spm\n\nspm.SentencePieceTrainer.train(\n    input='data.txt',\n    model_prefix='m',\n    vocab_size=8000,\n    model_type='bpe'\n)\n```\n\n**Training time**: ~1-2 minutes for 100MB corpus\n\n### Encode and decode\n\n```python\nimport sentencepiece as spm\n\n# Load model\nsp = spm.SentencePieceProcessor(model_file='m.model')\n\n# Encode to pieces\npieces = sp.encode('This is a test', out_type=str)\nprint(pieces)  # ['▁This', '▁is', '▁a', '▁test']\n\n# Encode to IDs\nids = sp.encode('This is a test', out_type=int)\nprint(ids)  # [284, 47, 11, 1243]\n\n# Decode\ntext = sp.decode(ids)\nprint(text)  # \"This is a test\"\n```\n\n## Language-independent design\n\n### Whitespace as symbol (▁)\n\n```python\ntext = \"Hello world\"\npieces = sp.encode(text, out_type=str)\nprint(pieces)  # ['▁Hello', '▁world']\n\n# Decode preserves spaces\ndecoded = sp.decode_pieces(pieces)\nprint(decoded)  # \"Hello world\"\n```\n\n**Key principle**: Treat text as raw Unicode, whitespace = ▁ (meta symbol)\n\n## Tokenization algorithms\n\n### BPE (Byte-Pair Encoding)\n\n```python\nspm.SentencePieceTrainer.train(\n    input='data.txt',\n    model_prefix='bpe_model',\n    vocab_size=16000,\n    model_type='bpe'\n)\n```\n\n**Used by**: mBART\n\n### Unigram (default)\n\n```python\nspm.SentencePieceTrainer.train(\n    input='data.txt',\n    model_prefix='unigram_model',\n    vocab_size=8000,\n    model_type='unigram'\n)\n```\n\n**Used by**: T5, ALBERT, XLNet\n\n## Training configuration\n\n### Essential parameters\n\n```python\nspm.SentencePieceTrainer.train(\n    input='corpus.txt',\n    model_prefix='m',\n    vocab_size=32000,\n    model_type='unigram',\n    character_coverage=0.9995,  # 1.0 for CJK\n    user_defined_symbols=['[SEP]', '[CLS]'],\n    unk_piece='\u003Cunk>',\n    num_threads=16\n)\n```\n\n### Character coverage\n\n| Language Type | Coverage | Rationale |\n|---------------|----------|-----------|\n| English       | 0.9995   | Most common chars |\n| CJK (Chinese) | 1.0      | All characters needed |\n| Multilingual  | 0.9995   | Balance |\n\n## Encoding options\n\n### Subword regularization\n\n```python\n# Sample different tokenizations\nfor _ in range(3):\n    pieces = sp.encode('tokenization', out_type=str, enable_sampling=True, alpha=0.1)\n    print(pieces)\n\n# Output (different each time):\n# ['▁token', 'ization']\n# ['▁tok', 'en', 'ization']\n```\n\n**Use case**: Data augmentation for robustness.\n\n## Common patterns\n\n### T5-style training\n\n```python\nspm.SentencePieceTrainer.train(\n    input='c4_corpus.txt',\n    model_prefix='t5',\n    vocab_size=32000,\n    model_type='unigram',\n    user_defined_symbols=[f'\u003Cextra_id_{i}>' for i in range(100)],\n    unk_id=2,\n    eos_id=1,\n    pad_id=0\n)\n```\n\n### Integration with transformers\n\n```python\nfrom transformers import T5Tokenizer\n\n# T5 uses SentencePiece internally\ntokenizer = T5Tokenizer.from_pretrained('t5-base')\ninputs = tokenizer('translate English to French: Hello', return_tensors='pt')\n```\n\n## Performance benchmarks\n\n### Training speed\n\n| Corpus | BPE (16k) | Unigram (8k) |\n|--------|-----------|--------------|\n| 100 MB | 1-2 min   | 3-4 min      |\n| 1 GB   | 10-15 min | 30-40 min    |\n\n### Tokenization speed\n\n- **SentencePiece**: 50,000 sentences\u002Fsec\n- **HF Tokenizers**: 200,000 sentences\u002Fsec (4× faster)\n\n## Supported models\n\n**T5 family**: `t5-base`, `t5-large` (32k vocab, Unigram)\n**ALBERT**: `albert-base-v2` (30k vocab, Unigram)\n**XLNet**: `xlnet-base-cased` (32k vocab, Unigram)\n**mBART**: `facebook\u002Fmbart-large-50` (250k vocab, BPE)\n\n## References\n\n- **[Training Guide](references\u002Ftraining.md)** - Detailed options, corpus preparation\n- **[Algorithms](references\u002Falgorithms.md)** - BPE vs Unigram, subword regularization\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fsentencepiece ⭐ 10,000+\n- **Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F1808.06226 (EMNLP 2018)\n- **Version**: 0.2.0+\n\n\n","# SentencePiece - Language-Independent Tokenization\n\nUnsupervised tokenizer that works on raw text without language-specific preprocessing.\n\n## When to use SentencePiece\n\n**Use SentencePiece when:**\n- Building multilingual models (no language-specific rules)\n- Working with CJK languages (Chinese, Japanese, Korean)\n- Need reproducible tokenization (deterministic vocabulary)\n- Want to train on raw text (no pre-tokenization needed)\n- Require lightweight deployment (6MB memory, 50k sentences\u002Fsec)\n\n**Performance**:\n- **Speed**: 50,000 sentences\u002Fsec\n- **Memory**: ~6MB for loaded model\n- **Languages**: All (language-independent)\n\n**Use alternatives instead**:\n- **HuggingFace Tokenizers**: Faster training, more flexibility\n- **tiktoken**: OpenAI models (GPT-3.5\u002F4)\n- **BERT WordPiece**: English-centric tasks\n\n## Quick start\n\n### Installation\n\n```bash\n# Python\npip install sentencepiece\n\n# C++ (requires CMake)\ngit clone https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fsentencepiece.git\ncd sentencepiece\nmkdir build && cd build\ncmake .. && make -j $(nproc)\nsudo make install\n```\n\n### Train model\n\n```bash\n# Command-line (BPE with 8000 vocab)\nspm_train --input=data.txt --model_prefix=m --vocab_size=8000 --model_type=bpe\n\n# Python API\nimport sentencepiece as spm\n\nspm.SentencePieceTrainer.train(\n    input='data.txt',\n    model_prefix='m',\n    vocab_size=8000,\n    model_type='bpe'\n)\n```\n\n**Training time**: ~1-2 minutes for 100MB corpus\n\n### Encode and decode\n\n```python\nimport sentencepiece as spm\n\n# Load model\nsp = spm.SentencePieceProcessor(model_file='m.model')\n\n# Encode to pieces\npieces = sp.encode('This is a test', out_type=str)\nprint(pieces)  # ['▁This', '▁is', '▁a', '▁test']\n\n# Encode to IDs\nids = sp.encode('This is a test', out_type=int)\nprint(ids)  # [284, 47, 11, 1243]\n\n# Decode\ntext = sp.decode(ids)\nprint(text)  # \"This is a test\"\n```\n\n## Language-independent design\n\n### Whitespace as symbol (▁)\n\n```python\ntext = \"Hello world\"\npieces = sp.encode(text, out_type=str)\nprint(pieces)  # ['▁Hello', '▁world']\n\n# Decode preserves spaces\ndecoded = sp.decode_pieces(pieces)\nprint(decoded)  # \"Hello world\"\n```\n\n**Key principle**: Treat text as raw Unicode, whitespace = ▁ (meta symbol)\n\n## Tokenization algorithms\n\n### BPE (Byte-Pair Encoding)\n\n```python\nspm.SentencePieceTrainer.train(\n    input='data.txt',\n    model_prefix='bpe_model',\n    vocab_size=16000,\n    model_type='bpe'\n)\n```\n\n**Used by**: mBART\n\n### Unigram (default)\n\n```python\nspm.SentencePieceTrainer.train(\n    input='data.txt',\n    model_prefix='unigram_model',\n    vocab_size=8000,\n    model_type='unigram'\n)\n```\n\n**Used by**: T5, ALBERT, XLNet\n\n## Training configuration\n\n### Essential parameters\n\n```python\nspm.SentencePieceTrainer.train(\n    input='corpus.txt',\n    model_prefix='m',\n    vocab_size=32000,\n    model_type='unigram',\n    character_coverage=0.9995,  # 1.0 for CJK\n    user_defined_symbols=['[SEP]', '[CLS]'],\n    unk_piece='\u003Cunk>',\n    num_threads=16\n)\n```\n\n### Character coverage\n\n| Language Type | Coverage | Rationale |\n|---------------|----------|-----------|\n| English       | 0.9995   | Most common chars |\n| CJK (Chinese) | 1.0      | All characters needed |\n| Multilingual  | 0.9995   | Balance |\n\n## Encoding options\n\n### Subword regularization\n\n```python\n# Sample different tokenizations\nfor _ in range(3):\n    pieces = sp.encode('tokenization', out_type=str, enable_sampling=True, alpha=0.1)\n    print(pieces)\n\n# Output (different each time):\n# ['▁token', 'ization']\n# ['▁tok', 'en', 'ization']\n```\n\n**Use case**: Data augmentation for robustness.\n\n## Common patterns\n\n### T5-style training\n\n```python\nspm.SentencePieceTrainer.train(\n    input='c4_corpus.txt',\n    model_prefix='t5',\n    vocab_size=32000,\n    model_type='unigram',\n    user_defined_symbols=[f'\u003Cextra_id_{i}>' for i in range(100)],\n    unk_id=2,\n    eos_id=1,\n    pad_id=0\n)\n```\n\n### Integration with transformers\n\n```python\nfrom transformers import T5Tokenizer\n\n# T5 uses SentencePiece internally\ntokenizer = T5Tokenizer.from_pretrained('t5-base')\ninputs = tokenizer('translate English to French: Hello', return_tensors='pt')\n```\n\n## Performance benchmarks\n\n### Training speed\n\n| Corpus | BPE (16k) | Unigram (8k) |\n|--------|-----------|--------------|\n| 100 MB | 1-2 min   | 3-4 min      |\n| 1 GB   | 10-15 min | 30-40 min    |\n\n### Tokenization speed\n\n- **SentencePiece**: 50,000 sentences\u002Fsec\n- **HF Tokenizers**: 200,000 sentences\u002Fsec (4× faster)\n\n## Supported models\n\n**T5 family**: `t5-base`, `t5-large` (32k vocab, Unigram)\n**ALBERT**: `albert-base-v2` (30k vocab, Unigram)\n**XLNet**: `xlnet-base-cased` (32k vocab, Unigram)\n**mBART**: `facebook\u002Fmbart-large-50` (250k vocab, BPE)\n\n## References\n\n- **[Training Guide](references\u002Ftraining.md)** - Detailed options, corpus preparation\n- **[Algorithms](references\u002Falgorithms.md)** - BPE vs Unigram, subword regularization\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fsentencepiece ⭐ 10,000+\n- **Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F1808.06226 (EMNLP 2018)\n- **Version**: 0.2.0+\n\n\n",{"name":5,"tags":24,"author":35,"license":15,"version":36,"description":7,"dependencies":37},[25,26,27,28,29,30,31,32,33,34],"Tokenization","SentencePiece","Language-Independent","BPE","Unigram","Multilingual","CJK Languages","Unicode","Deterministic","Google","Orchestra Research","1.0.0",[5,38],"transformers","skills","4fb202b97f2676132ac8713dbd2370cefacacb81",".agents\u002Fskills\u002Fsentencepiece","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills","https:\u002F\u002Fgithub.com\u002Fzengrong233","https:\u002F\u002Fwtu.edu.cn\u002F","my name songzengrong\r\n  research interest's:NLP","2026-06-04 10:48:22",{"source":8,"skillName":5,"commands":48,"prompt":58,"zipAvailable":59},[49,52,55],{"tool":50,"command":51},"npx","npx skills add zengrong233\u002Fdevin-skills --skill sentencepiece",{"tool":53,"command":54},"bunx","bunx skills add zengrong233\u002Fdevin-skills --skill sentencepiece",{"tool":56,"command":57},"pnpm","pnpm dlx skills add zengrong233\u002Fdevin-skills --skill sentencepiece","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fsentencepiece\u002FSKILL.md",true,[61,67,71],{"path":62,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":65,"sourceHtmlUrl":66},".openskills.json","REFERENCE","参考文档",235,"https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fsentencepiece\u002F.openskills.json",{"path":68,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":69,"sourceHtmlUrl":70},"references\u002Falgorithms.md",4212,"https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fsentencepiece\u002Freferences\u002Falgorithms.md",{"path":72,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":73,"sourceHtmlUrl":74},"references\u002Ftraining.md",6240,"https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fsentencepiece\u002Freferences\u002Ftraining.md",[76,83,90,99,107,114],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":19},"01M759188DV0S9ARY1ETHCEY45","whisper","OpenAI's general-purpose speech recognition model. Supports 99 languages, transcription, translation to English, and language identification. Six model sizes from tiny (39M params) to large (1550M params). Use for speech-to-text, podcast transcription, or multilingual audio processing. Best for robust, multilingual ASR.","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fwhisper\u002FSKILL.md",[],[],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":19},"01M75913ZB4EDVWYGYRTRWTKD1","weights-and-biases","Track ML experiments with automatic logging, visualize training in real-time, optimize hyperparameters with sweeps, and manage model registry with W&B - collaborative MLOps platform","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fweights-and-biases\u002FSKILL.md",[],[],{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":94,"licenseFileUrl":95,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":19},"01M75910YNW7H0KZQDR96TGWZ9","webapp-testing","Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fwebapp-testing\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fwebapp-testing\u002FSKILL.md",[],[],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":94,"licenseFileUrl":103,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":19},"01M7590X6XJR78KVSHCVFYB59A","web-artifacts-builder","Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn\u002Fui). Use for complex artifacts requiring state management, routing, or shadcn\u002Fui components - not for simple single-file HTML\u002FJSX artifacts.","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fweb-artifacts-builder\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fweb-artifacts-builder\u002FSKILL.md",[],[],{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":111,"categories":112,"keywords":113,"publishedAt":19},"01M7590RENMJJTXV44P8G553ZQ","serving-llms-vllm","Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency\u002Fthroughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ\u002FAWQ\u002FFP8), and tensor parallelism.","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fvllm\u002FSKILL.md",[],[],{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":118,"categories":119,"keywords":120,"publishedAt":19},"01M7590GA6EZKJA8GAAMMPAXBB","verl-rl-training","Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fverl\u002FSKILL.md",[],[]]