[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7EW7CFXYB32SPPD5X1E8281":3,"skill-same:01M7EW7CFXYB32SPPD5X1E8281":47},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":28,"repositoryHtmlUrl":29,"repositoryHomepageUrl":6,"ownerHtmlUrl":30,"ownerBlogUrl":31,"ownerDescription":6,"updatedAt":32,"install":33,"resources":46},"01M7EW7CFXYB32SPPD5X1E8281","tokenizers",null,"Understand and work with tokenizers — BPE, WordPiece, Unigram, special tokens, and the practical effects of tokenization on cost and quality. Use when token counts, chunking, or model inputs misbehave.","TanNhatCMS\u002Fjarvis","TanNhatCMS","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F113796420?v=4","01M753MV17KG5CD3152G3WQEQY","KietNT","01M7EKEDYVYQ6PWVJJMZ911VMX",0,"MIT","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fams-tokenizers\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: tokenizers\ndescription: Understand and work with tokenizers — BPE, WordPiece, Unigram, special tokens, and the practical effects of tokenization on cost and quality. Use when token counts, chunking, or model inputs misbehave.\ncategory: ai-research\n---\n\n# Tokenizers\n\nTokenizers turn text into the integer sequences models actually process. They're invisible until \nthey cause problems: surprising costs, broken chunking, or corrupted inputs. Understanding them \nturns those mysteries into mechanics.\n\n## Overview\n\nModern tokenizers learn subword vocabularies from data (BPE merges frequent pairs; WordPiece and \nUnigram have their own strategies), then split text into those pieces. Key consequences: token \ncount ≠ word count (varies by language and content — code and non-English text tokenize less \nefficiently), special tokens mark boundaries and roles, and detokenization isn't always the exact \ninverse. Every model has its own tokenizer — using the wrong one corrupts inputs silently.\n\n## When to use\n\n- Estimating costs: tokens are the billing unit — count with the right tokenizer.\n- Debugging \"the model saw something different\": encoding issues, truncation surprises.\n- Designing chunking for RAG: chunk by tokens, not characters.\n- Working with structured formats: JSON, code, and special tokens need care.\n\n## Core concepts\n\n- **Subword algorithms**: BPE (merge frequent pairs), WordPiece (likelihood-based merges), Unigram \n(probabilistic pruning). Different trade-offs; what matters is which one your model uses.\n- **Vocabulary**: the learned piece inventory, typically 30k–200k entries. Larger vocabs tokenize \nmore efficiently (fewer tokens per text) but need bigger embedding tables.\n- **Special tokens**: BOS\u002FEOS markers, padding, separators, and chat-template tokens (`\u003C|user|>`, \netc.). Chat templates are tokenizer-level — apply them via the tokenizer, not by hand.\n- **Fertility**: tokens per word — varies wildly by language and domain. English prose ~1.3; code \nhigher; some languages 2–3×. Budget multilingual contexts accordingly.\n- **Encode\u002Fdecode round-trip**: decoding isn't always exact (whitespace handling, byte fallbacks). \nTest round-trips for pipelines that reconstruct text.\n- **Pre-tokenization**: how text is split before BPE (whitespace, punctuation rules). Explains many \n\"why did it split there?\" mysteries.\n\n## Practical workflow\n\n1. Identify the model's actual tokenizer — never assume; load it from the model's own artifacts.\n2. For cost estimation: tokenize representative inputs with that tokenizer, not a generic counter.\n3. For chunking: chunk on token boundaries with the model's tokenizer; account for special tokens \nadded by chat templates.\n4. When debugging inputs: decode what the model actually received — compare against what you \nintended.\n5. Apply chat templates through the tokenizer's template support; verify the rendered format \nmatches documentation.\n6. For multilingual work: measure fertility per language; adjust context budgets per language.\n\n```text\nTokenizer debugging checklist:\n[ ] Using the model's own tokenizer (not a generic one)\n[ ] Chat template applied via tokenizer, output verified\n[ ] Token counts measured on real inputs per language\n[ ] Special tokens accounted for in budgets\n[ ] Round-trip tested for reconstruction pipelines\n```\n\n## Common pitfalls\n\n- **Wrong tokenizer**: counting or chunking with a different tokenizer than the model uses. \nSilently wrong everywhere.\n- **Hand-rolled chat templates**: formatting `\u003C|user|>` blocks manually and getting subtle details \nwrong. Use the tokenizer's template.\n- **Character-based chunking**: splitting mid-token or mid-word for RAG. Chunk by tokens with \noverlap.\n- **Ignoring fertility**: budgeting English token counts for multilingual content. Measure per \nlanguage.\n- **Truncation surprises**: tokenizers truncate silently at limits. Check lengths; truncate \ndeliberately with the right strategy (head, tail, or middle).\n- **Byte-fallback blindness**: rare characters becoming multi-token byte sequences. Affects cost \nand sometimes quality for unusual scripts.\n","# Tokenizers\n\nTokenizers turn text into the integer sequences models actually process. They're invisible until \nthey cause problems: surprising costs, broken chunking, or corrupted inputs. Understanding them \nturns those mysteries into mechanics.\n\n## Overview\n\nModern tokenizers learn subword vocabularies from data (BPE merges frequent pairs; WordPiece and \nUnigram have their own strategies), then split text into those pieces. Key consequences: token \ncount ≠ word count (varies by language and content — code and non-English text tokenize less \nefficiently), special tokens mark boundaries and roles, and detokenization isn't always the exact \ninverse. Every model has its own tokenizer — using the wrong one corrupts inputs silently.\n\n## When to use\n\n- Estimating costs: tokens are the billing unit — count with the right tokenizer.\n- Debugging \"the model saw something different\": encoding issues, truncation surprises.\n- Designing chunking for RAG: chunk by tokens, not characters.\n- Working with structured formats: JSON, code, and special tokens need care.\n\n## Core concepts\n\n- **Subword algorithms**: BPE (merge frequent pairs), WordPiece (likelihood-based merges), Unigram \n(probabilistic pruning). Different trade-offs; what matters is which one your model uses.\n- **Vocabulary**: the learned piece inventory, typically 30k–200k entries. Larger vocabs tokenize \nmore efficiently (fewer tokens per text) but need bigger embedding tables.\n- **Special tokens**: BOS\u002FEOS markers, padding, separators, and chat-template tokens (`\u003C|user|>`, \netc.). Chat templates are tokenizer-level — apply them via the tokenizer, not by hand.\n- **Fertility**: tokens per word — varies wildly by language and domain. English prose ~1.3; code \nhigher; some languages 2–3×. Budget multilingual contexts accordingly.\n- **Encode\u002Fdecode round-trip**: decoding isn't always exact (whitespace handling, byte fallbacks). \nTest round-trips for pipelines that reconstruct text.\n- **Pre-tokenization**: how text is split before BPE (whitespace, punctuation rules). Explains many \n\"why did it split there?\" mysteries.\n\n## Practical workflow\n\n1. Identify the model's actual tokenizer — never assume; load it from the model's own artifacts.\n2. For cost estimation: tokenize representative inputs with that tokenizer, not a generic counter.\n3. For chunking: chunk on token boundaries with the model's tokenizer; account for special tokens \nadded by chat templates.\n4. When debugging inputs: decode what the model actually received — compare against what you \nintended.\n5. Apply chat templates through the tokenizer's template support; verify the rendered format \nmatches documentation.\n6. For multilingual work: measure fertility per language; adjust context budgets per language.\n\n```text\nTokenizer debugging checklist:\n[ ] Using the model's own tokenizer (not a generic one)\n[ ] Chat template applied via tokenizer, output verified\n[ ] Token counts measured on real inputs per language\n[ ] Special tokens accounted for in budgets\n[ ] Round-trip tested for reconstruction pipelines\n```\n\n## Common pitfalls\n\n- **Wrong tokenizer**: counting or chunking with a different tokenizer than the model uses. \nSilently wrong everywhere.\n- **Hand-rolled chat templates**: formatting `\u003C|user|>` blocks manually and getting subtle details \nwrong. Use the tokenizer's template.\n- **Character-based chunking**: splitting mid-token or mid-word for RAG. Chunk by tokens with \noverlap.\n- **Ignoring fertility**: budgeting English token counts for multilingual content. Measure per \nlanguage.\n- **Truncation surprises**: tokenizers truncate silently at limits. Check lengths; truncate \ndeliberately with the right strategy (head, tail, or middle).\n- **Byte-fallback blindness**: rare characters becoming multi-token byte sequences. Affects cost \nand sometimes quality for unusual scripts.\n",{"name":5,"category":24,"description":7},"ai-research","main","af388a77c2fe5e0b4fd879cca4300a0ffa9a6579","skills\u002Fams-tokenizers",1,"https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis","https:\u002F\u002Fgithub.com\u002FTanNhatCMS","https:\u002F\u002Fmrkiet.name.vn","2026-11-07 14:23:08",{"source":8,"skillName":5,"commands":34,"prompt":44,"zipAvailable":45},[35,38,41],{"tool":36,"command":37},"npx","npx skills add TanNhatCMS\u002Fjarvis --skill tokenizers",{"tool":39,"command":40},"bunx","bunx skills add TanNhatCMS\u002Fjarvis --skill tokenizers",{"tool":42,"command":43},"pnpm","pnpm dlx skills add TanNhatCMS\u002Fjarvis --skill tokenizers","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fams-tokenizers\u002FSKILL.md",true,[],[48,55,63,70,77,84],{"skillNo":49,"name":50,"nameZh":6,"description":51,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":52,"categories":53,"keywords":54,"publishedAt":19},"01M7EWYJR0J8DARY7E59Z6Y9MV","web-scrape","Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open\u002Ffetch fails on JS-heavy pages, or when you need clean markdown\u002Fstructured extraction from a site. Requires FIRECRAWL_API_KEY.","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fweb-scrape\u002FSKILL.md",[],[],{"skillNo":56,"name":57,"nameZh":6,"description":58,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":59,"sourceHtmlUrl":60,"categories":61,"keywords":62,"publishedAt":19},"01M7EWYBN5P6F2AJG3FR24T9BP","design-taste-frontend","tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML\u002FCSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, \"AI-made\", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Ftaste-design\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Ftaste-design\u002FSKILL.md",[],[],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":19},"01M7EWY8FQFNQ2GB7NZZWGSSDX","test-driven-development","Use when implementing any feature or bugfix, before writing implementation code","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-tdd\u002FSKILL.md",[],[],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M7EWY5SHBFRH1HDHVWM9K7TV","systematic-debugging","Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-debugging\u002FSKILL.md",[],[],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":19},"01M7EWY3NM0Z0JAKS7HDZ24K66","requesting-code-review","Use when completing tasks, implementing major features, or before merging to verify work meets requirements","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-code-review\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":19},"01M7EWY0WRMWX4M9ZFY0WKC593","brainstorming","You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-brainstorming\u002FSKILL.md",[],[]]