[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6ACF7V56CCWHNVR614VBNX5":3,"skill-same:01M6ACF7V56CCWHNVR614VBNX5":87},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":33,"linkOnly":34,"markdownContent":35,"markdownBody":36,"frontMatter":37,"refName":38,"commitSha":39,"skillDirectory":40,"forksCount":41,"repositoryHtmlUrl":42,"repositoryHomepageUrl":43,"ownerHtmlUrl":44,"ownerBlogUrl":45,"ownerDescription":6,"updatedAt":46,"install":47,"resources":60},"01M6ACF7V56CCWHNVR614VBNX5","tika-eval-encoding-regression",null,"Condensed tika-eval pattern for charset-detector regression hunts (\"A picks encoding X, B picks Y\") using one build and two configs — encoding-pair flip queries, OOV\u002Flanguageness\u002FFFFD signals, per-file detector attribution.","apache\u002Ftika","apache","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F47359?v=4","01M6AC83E6K4FDKK7HZ8WNMQKE","The Apache Software Foundation","01M6ACEMPJD6J74AGZBFFEGEYV",4145,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fapache\u002Ftika\u002Fblob\u002Fmain\u002F.skills\u002Fdevs\u002Ftika-eval-encoding-regression\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32],"tika","tika eval encoding regression","encoding","regression","eval","file","attribution","pair","queries","build","pattern","signals","detector","for","2026-11-10 10:12:30",false,"---\nname: tika-eval-encoding-regression\ndescription: >\n  Condensed tika-eval pattern for charset-detector regression hunts (\"A picks\n  encoding X, B picks Y\") using one build and two configs — encoding-pair\n  flip queries, OOV\u002Flanguageness\u002FFFFD signals, per-file detector attribution.\n---\n\n\u003C!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license agreements.  See the NOTICE file distributed with\nthis work for additional information regarding copyright ownership.\nThe ASF licenses this file to You under the Apache License, Version 2.0\n(the \"License\"); you may not use this file except in compliance with\nthe License.  You may obtain a copy of the License at\n\n    http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n-->\n\nLocal override: `$TIKA_SKILLS_LOCAL\u002Ftika-eval-encoding-regression\u002FLOCAL.md` (default `~\u002F.tika-skills`),\nread after this file, wins on conflict.\n\n# tika-eval for encoding-detector regression hunts\n\nA condensed pattern for finding SBCS→CJK style charset-detector regressions\n(or any \"A picks encoding X, B picks encoding Y\" question) without\nbuilding two tika-app distributions.\n\n## Two configs, one build\n\nEncoding-detector experiments don't need a \"before\" and \"after\" tika-app —\nthe chain composition is per-config. Run the SAME tika-app twice against\ntwo configs, treat the outputs as `-a` and `-b`. Much faster than\n`tika-eval-compare`'s two-build flow.\n\n```bash\n# build once\n.\u002Fmvnw clean install -pl tika-app -am -Pfast -DskipTests \\\n  -Dmaven.repo.local=$(pwd)\u002F.local_m2_repo\nunzip -q tika-app\u002Ftarget\u002Ftika-app-*.zip -d \u002Ftmp\u002Ftika-app-current\n\n# two configs (any combination of detectors)\njava -jar \u002Ftmp\u002Ftika-app-current\u002Ftika-app-*.jar \\\n  --config=tika-config-3x-default.json \\\n  -i \u003Ccorpus> -o \u003Cworkdir>\u002Fextracts\u002FA -n 6\njava -jar \u002Ftmp\u002Ftika-app-current\u002Ftika-app-*.jar \\\n  --config=tika-config-junkfilter-combiner.json \\\n  -i \u003Ccorpus> -o \u003Cworkdir>\u002Fextracts\u002FB -n 6\n\n# normal Compare\njava -jar \u002Ftmp\u002Ftika-eval-current\u002Ftika-eval-app-*.jar Compare \\\n  -a \u003Cworkdir>\u002Fextracts\u002FA -b \u003Cworkdir>\u002Fextracts\u002FB -d \u003Cworkdir>\u002Fextracts\u002FA-vs-B -r -rd \u003Cworkdir>\u002Fextracts\u002FA-vs-B-reports\n```\n\n### Canonical 3.x-default encoding chain config\n\n```json\n{\n  \"encoding-detectors\": [\n    {\"html-encoding-detector\": {}},\n    {\"universal-encoding-detector\": {}},\n    {\"icu4j-encoding-detector\": {}}\n  ]\n}\n```\n\n### Canonical 4.x junkfilter chain config\n\n```json\n{\n  \"encoding-detectors\": [\n    {\"bom-detector\": {}},\n    {\"html-encoding-detector\": {}},\n    {\"mojibuster-encoding-detector\": {}},\n    {\"junk-filter-encoding-detector\": {}}\n  ]\n}\n```\n\n### Per-detector isolation configs\n\nEach detector wired alone lives in `\u003Cworkdir>\u002Fconfigs\u002F`:\n`tika-config-bom.json`, `tika-config-html.json`, `tika-config-htmlstandard.json`,\n`tika-config-universal.json`, `tika-config-icu4j.json`,\n`tika-config-mojibuster.json`, `tika-config-junkfilter-chain.json`.\nUse these for chain-attribution work (which detector did the detection).\n\n## Encoding-pair flip query\n\n`MIMES.MIME_STRING` for text-y mimes is `text\u002Fhtml; charset=X` form. Extract\nthe charset with a regex split, group by `(enc_a, enc_b)`, filter pairs.\nA=before\u002F`-a`, B=after\u002F`-b`; join on `pa.ID = pb.ID` (paired by id).\n\n```sql\nSELECT\n  REGEXP_REPLACE(ma.MIME_STRING, '^.*charset=', '') AS enc_a,\n  REGEXP_REPLACE(mb.MIME_STRING, '^.*charset=', '') AS enc_b,\n  COUNT(*) n,\n  SUM(cb.NUM_COMMON_TOKENS - ca.NUM_COMMON_TOKENS) AS delta_common\nFROM PROFILES_A pa\nJOIN PROFILES_B pb ON pa.ID = pb.ID\nJOIN MIMES ma ON pa.MIME_ID = ma.MIME_ID\nJOIN MIMES mb ON pb.MIME_ID = mb.MIME_ID\nJOIN CONTENTS_A ca ON ca.ID = pa.ID\nJOIN CONTENTS_B cb ON cb.ID = pb.ID\nWHERE ma.MIME_STRING LIKE '%charset=%' AND mb.MIME_STRING LIKE '%charset=%'\n  AND REGEXP_REPLACE(ma.MIME_STRING, '^.*charset=', '') \u003C>\n      REGEXP_REPLACE(mb.MIME_STRING, '^.*charset=', '')\nGROUP BY enc_a, enc_b\nORDER BY n DESC, delta_common ASC LIMIT 50;\n```\n\nAdd an `IN (...)` filter on either side to constrain to a family\n(e.g. SBCS-Western → CJK):\n\n```sql\n  AND REGEXP_REPLACE(ma.MIME_STRING,'^.*charset=','')\n      IN ('windows-1252','ISO-8859-1','ISO-8859-15','ISO-8859-2','ISO-8859-3',\n          'windows-1250','windows-1254','windows-1257','ISO-8859-13',\n          'windows-1258','x-MacRoman','IBM850','IBM852')\n  AND REGEXP_REPLACE(mb.MIME_STRING,'^.*charset=','')\n      IN ('GB18030','GBK','GB2312','Big5','Big5-HKSCS','Shift_JIS','EUC-JP',\n          'EUC-KR','x-EUC-TW','x-windows-874','x-windows-949',\n          'ISO-2022-JP','ISO-2022-KR','ISO-2022-CN')\n```\n\n### Per-file drilldown\n\nJoin `CONTAINERS` to get the source path; pull `LANG_ID_1` from both sides\nto see whether language detection agrees the content is Western while the\ncharset has flipped to CJK (the regression's defining shape):\n\n```sql\nSELECT ct.FILE_PATH,\n       REGEXP_REPLACE(ma.MIME_STRING,'^.*charset=','') AS enc_a,\n       REGEXP_REPLACE(mb.MIME_STRING,'^.*charset=','') AS enc_b,\n       ca.NUM_COMMON_TOKENS AS ca_tok, cb.NUM_COMMON_TOKENS AS cb_tok,\n       cb.NUM_COMMON_TOKENS - ca.NUM_COMMON_TOKENS AS delta,\n       ca.LANG_ID_1 AS lang_a, cb.LANG_ID_1 AS lang_b\nFROM PROFILES_A pa JOIN PROFILES_B pb ON pa.ID = pb.ID\nJOIN MIMES ma ON pa.MIME_ID = ma.MIME_ID JOIN MIMES mb ON pb.MIME_ID = mb.MIME_ID\nJOIN CONTENTS_A ca ON ca.ID = pa.ID JOIN CONTENTS_B cb ON cb.ID = pb.ID\nJOIN CONTAINERS ct ON ct.CONTAINER_ID = pa.CONTAINER_ID\nWHERE \u003Cenc_a\u002Fenc_b filter as above>\nORDER BY delta ASC LIMIT 15;\n```\n\n## Reading the signals — OOV, languageness, and FFFD together\n\nNo single signal is authoritative. Use `oov` as a **secondary** signal alongside\n`languageness` (the junk-model coherence z-score) and the U+FFFD rate — each is\nright where the others are blind, so cross-check rather than ranking on any one.\n(Established 2026-06-03: a 40-file OOV-\"worse\" set was mostly metric artifacts\nonce languageness\u002FFFFD were brought in — only ~6 were real. But OOV is also the\n*correct* signal where languageness is blind, so neither dominates.)\n\n- **OOV can mislead** when langid shifts — a CJK\u002FUTF-8 recovery in B is scored\n  against a different vocab → higher OOV though B is right — or when a wrong\n  decode fragments words into more short common tokens (→ higher count for the\n  WORSE decode). A common-token delta is a signal, not proof.\n- **languageness can mislead** on SBCS↔SBCS cross-script mojibake — Greek decoded\n  as KOI8-R is \"coherent\" Cyrillic, so `languageness` stays flat while `oov`\n  correctly flags it. Conversely languageness catches OOV's CJK\u002Fscript-recovery\n  blind spot. Each covers the other's blind spot.\n- **FFFD rate** flags decode failures (illegal bytes): `num_replacement \u002F\n  num_non_ascii` (un-diluted; `\u002F content_length` dilutes to ~0 on ASCII-heavy\n  docs). Tika strips C0 controls at extraction, so legal-but-wrong (C1) mojibake\n  does not surface here — that signal belongs in the detector chain, not the eval.\n- **In practice:** when the signals agree, high confidence; when they disagree\n  (OOV-worse but languageness-better, or vice versa), that file needs a look —\n  the disagreement points you at WHICH files to inspect, it does not by itself\n  declare OOV or languageness \"wrong.\" Split OOV-worse by languageness direction\n  (query in `tika-eval-regression.adoc`).\n\n### Isolate a change against the PRIOR run, not just 3.x\n\nTo see what one chain change actually did, Compare the new run against the\n*previous* 4.x run (B-new vs B-prior), not only vs 3.x. The diff should be\n*surgical* — e.g. the within-Latin letter gate moved exactly 6 files\n(IBM850 \u002F x-MacRoman → windows-1252) vs the prior run and nothing else. A\nbigger-than-expected diff means the change fired more broadly than intended.\n\n## Per-file detector attribution (`tk:encoding-detection-trace`)\n\nEvery JSON extract from a chain with multiple detectors carries\n`tk:encoding-detection-trace` in metadata. It's a per-detector emission\nlog with the META detector's arbitration tag at the end:\n\n```\nMojibusterEncodingDetector->Shift_JIS[STATISTICAL](1.00) [junk-filter-selected]\n```\n\nWhen investigating \"why did B pick X for this file?\", read this trace first\n— it tells you which base detector(s) emitted candidates and which one the\nmeta detector chose. If the trace shows ONLY Mojibuster firing with a CJK\npick, the bug is in Mojibuster's emission (pool too narrow), not in\nJunkFilter's arbitration.\n\n`tk:encoding-detector` is the simple-name credit string;\n`tk:detected-encoding` is the final answer (also in `Content-Encoding`).\n\n## Reproducing a single-file detection without a full chain\n\n```bash\n.\u002Fmvnw -q -pl tika-ml\u002Ftika-ml-junkdetect -Dmaven.repo.local=$(pwd)\u002F.local_m2_repo \\\n  -Dexec.classpathScope=test \\\n  -Dexec.mainClass=org.apache.tika.ml.junkdetect.TraceJunkFilter \\\n  -Dexec.args=\"--file \u003Cpath> --auto-candidates --content-cleaner --head-bytes 524288 --sample 120\" \\\n  exec:java\n```\n\nKey flags:\n- `--auto-candidates` — use Mojibuster's per-file pool as the candidate set\n- `--content-cleaner` — decode each candidate then run text through\n  `HtmlContentCleaner` to match the live chain\n- `--head-bytes 524288` — read up to 512 KB raw to match\n  `AdaptiveProbe.DEFAULT_RAW_CAP`. The default `READ_LIMIT` of 16 KB will\n  give a *different* probe than the live chain on long markup-heavy pages\n  and lead you to disagree with the live chain's pick. Always pass this\n  when reconciling a TraceJunkFilter run with a live extract.\n\nWithout `--head-bytes`, you are looking at a different probe than the\nchain saw — this is the most common source of \"trace says X, chain\nsays Y\" confusion.\n","\u003C!--\nLicensed to the Apache Software Foundation (ASF) under one or more\ncontributor license agreements.  See the NOTICE file distributed with\nthis work for additional information regarding copyright ownership.\nThe ASF licenses this file to You under the Apache License, Version 2.0\n(the \"License\"); you may not use this file except in compliance with\nthe License.  You may obtain a copy of the License at\n\n    http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n-->\n\nLocal override: `$TIKA_SKILLS_LOCAL\u002Ftika-eval-encoding-regression\u002FLOCAL.md` (default `~\u002F.tika-skills`),\nread after this file, wins on conflict.\n\n# tika-eval for encoding-detector regression hunts\n\nA condensed pattern for finding SBCS→CJK style charset-detector regressions\n(or any \"A picks encoding X, B picks encoding Y\" question) without\nbuilding two tika-app distributions.\n\n## Two configs, one build\n\nEncoding-detector experiments don't need a \"before\" and \"after\" tika-app —\nthe chain composition is per-config. Run the SAME tika-app twice against\ntwo configs, treat the outputs as `-a` and `-b`. Much faster than\n`tika-eval-compare`'s two-build flow.\n\n```bash\n# build once\n.\u002Fmvnw clean install -pl tika-app -am -Pfast -DskipTests \\\n  -Dmaven.repo.local=$(pwd)\u002F.local_m2_repo\nunzip -q tika-app\u002Ftarget\u002Ftika-app-*.zip -d \u002Ftmp\u002Ftika-app-current\n\n# two configs (any combination of detectors)\njava -jar \u002Ftmp\u002Ftika-app-current\u002Ftika-app-*.jar \\\n  --config=tika-config-3x-default.json \\\n  -i \u003Ccorpus> -o \u003Cworkdir>\u002Fextracts\u002FA -n 6\njava -jar \u002Ftmp\u002Ftika-app-current\u002Ftika-app-*.jar \\\n  --config=tika-config-junkfilter-combiner.json \\\n  -i \u003Ccorpus> -o \u003Cworkdir>\u002Fextracts\u002FB -n 6\n\n# normal Compare\njava -jar \u002Ftmp\u002Ftika-eval-current\u002Ftika-eval-app-*.jar Compare \\\n  -a \u003Cworkdir>\u002Fextracts\u002FA -b \u003Cworkdir>\u002Fextracts\u002FB -d \u003Cworkdir>\u002Fextracts\u002FA-vs-B -r -rd \u003Cworkdir>\u002Fextracts\u002FA-vs-B-reports\n```\n\n### Canonical 3.x-default encoding chain config\n\n```json\n{\n  \"encoding-detectors\": [\n    {\"html-encoding-detector\": {}},\n    {\"universal-encoding-detector\": {}},\n    {\"icu4j-encoding-detector\": {}}\n  ]\n}\n```\n\n### Canonical 4.x junkfilter chain config\n\n```json\n{\n  \"encoding-detectors\": [\n    {\"bom-detector\": {}},\n    {\"html-encoding-detector\": {}},\n    {\"mojibuster-encoding-detector\": {}},\n    {\"junk-filter-encoding-detector\": {}}\n  ]\n}\n```\n\n### Per-detector isolation configs\n\nEach detector wired alone lives in `\u003Cworkdir>\u002Fconfigs\u002F`:\n`tika-config-bom.json`, `tika-config-html.json`, `tika-config-htmlstandard.json`,\n`tika-config-universal.json`, `tika-config-icu4j.json`,\n`tika-config-mojibuster.json`, `tika-config-junkfilter-chain.json`.\nUse these for chain-attribution work (which detector did the detection).\n\n## Encoding-pair flip query\n\n`MIMES.MIME_STRING` for text-y mimes is `text\u002Fhtml; charset=X` form. Extract\nthe charset with a regex split, group by `(enc_a, enc_b)`, filter pairs.\nA=before\u002F`-a`, B=after\u002F`-b`; join on `pa.ID = pb.ID` (paired by id).\n\n```sql\nSELECT\n  REGEXP_REPLACE(ma.MIME_STRING, '^.*charset=', '') AS enc_a,\n  REGEXP_REPLACE(mb.MIME_STRING, '^.*charset=', '') AS enc_b,\n  COUNT(*) n,\n  SUM(cb.NUM_COMMON_TOKENS - ca.NUM_COMMON_TOKENS) AS delta_common\nFROM PROFILES_A pa\nJOIN PROFILES_B pb ON pa.ID = pb.ID\nJOIN MIMES ma ON pa.MIME_ID = ma.MIME_ID\nJOIN MIMES mb ON pb.MIME_ID = mb.MIME_ID\nJOIN CONTENTS_A ca ON ca.ID = pa.ID\nJOIN CONTENTS_B cb ON cb.ID = pb.ID\nWHERE ma.MIME_STRING LIKE '%charset=%' AND mb.MIME_STRING LIKE '%charset=%'\n  AND REGEXP_REPLACE(ma.MIME_STRING, '^.*charset=', '') \u003C>\n      REGEXP_REPLACE(mb.MIME_STRING, '^.*charset=', '')\nGROUP BY enc_a, enc_b\nORDER BY n DESC, delta_common ASC LIMIT 50;\n```\n\nAdd an `IN (...)` filter on either side to constrain to a family\n(e.g. SBCS-Western → CJK):\n\n```sql\n  AND REGEXP_REPLACE(ma.MIME_STRING,'^.*charset=','')\n      IN ('windows-1252','ISO-8859-1','ISO-8859-15','ISO-8859-2','ISO-8859-3',\n          'windows-1250','windows-1254','windows-1257','ISO-8859-13',\n          'windows-1258','x-MacRoman','IBM850','IBM852')\n  AND REGEXP_REPLACE(mb.MIME_STRING,'^.*charset=','')\n      IN ('GB18030','GBK','GB2312','Big5','Big5-HKSCS','Shift_JIS','EUC-JP',\n          'EUC-KR','x-EUC-TW','x-windows-874','x-windows-949',\n          'ISO-2022-JP','ISO-2022-KR','ISO-2022-CN')\n```\n\n### Per-file drilldown\n\nJoin `CONTAINERS` to get the source path; pull `LANG_ID_1` from both sides\nto see whether language detection agrees the content is Western while the\ncharset has flipped to CJK (the regression's defining shape):\n\n```sql\nSELECT ct.FILE_PATH,\n       REGEXP_REPLACE(ma.MIME_STRING,'^.*charset=','') AS enc_a,\n       REGEXP_REPLACE(mb.MIME_STRING,'^.*charset=','') AS enc_b,\n       ca.NUM_COMMON_TOKENS AS ca_tok, cb.NUM_COMMON_TOKENS AS cb_tok,\n       cb.NUM_COMMON_TOKENS - ca.NUM_COMMON_TOKENS AS delta,\n       ca.LANG_ID_1 AS lang_a, cb.LANG_ID_1 AS lang_b\nFROM PROFILES_A pa JOIN PROFILES_B pb ON pa.ID = pb.ID\nJOIN MIMES ma ON pa.MIME_ID = ma.MIME_ID JOIN MIMES mb ON pb.MIME_ID = mb.MIME_ID\nJOIN CONTENTS_A ca ON ca.ID = pa.ID JOIN CONTENTS_B cb ON cb.ID = pb.ID\nJOIN CONTAINERS ct ON ct.CONTAINER_ID = pa.CONTAINER_ID\nWHERE \u003Cenc_a\u002Fenc_b filter as above>\nORDER BY delta ASC LIMIT 15;\n```\n\n## Reading the signals — OOV, languageness, and FFFD together\n\nNo single signal is authoritative. Use `oov` as a **secondary** signal alongside\n`languageness` (the junk-model coherence z-score) and the U+FFFD rate — each is\nright where the others are blind, so cross-check rather than ranking on any one.\n(Established 2026-06-03: a 40-file OOV-\"worse\" set was mostly metric artifacts\nonce languageness\u002FFFFD were brought in — only ~6 were real. But OOV is also the\n*correct* signal where languageness is blind, so neither dominates.)\n\n- **OOV can mislead** when langid shifts — a CJK\u002FUTF-8 recovery in B is scored\n  against a different vocab → higher OOV though B is right — or when a wrong\n  decode fragments words into more short common tokens (→ higher count for the\n  WORSE decode). A common-token delta is a signal, not proof.\n- **languageness can mislead** on SBCS↔SBCS cross-script mojibake — Greek decoded\n  as KOI8-R is \"coherent\" Cyrillic, so `languageness` stays flat while `oov`\n  correctly flags it. Conversely languageness catches OOV's CJK\u002Fscript-recovery\n  blind spot. Each covers the other's blind spot.\n- **FFFD rate** flags decode failures (illegal bytes): `num_replacement \u002F\n  num_non_ascii` (un-diluted; `\u002F content_length` dilutes to ~0 on ASCII-heavy\n  docs). Tika strips C0 controls at extraction, so legal-but-wrong (C1) mojibake\n  does not surface here — that signal belongs in the detector chain, not the eval.\n- **In practice:** when the signals agree, high confidence; when they disagree\n  (OOV-worse but languageness-better, or vice versa), that file needs a look —\n  the disagreement points you at WHICH files to inspect, it does not by itself\n  declare OOV or languageness \"wrong.\" Split OOV-worse by languageness direction\n  (query in `tika-eval-regression.adoc`).\n\n### Isolate a change against the PRIOR run, not just 3.x\n\nTo see what one chain change actually did, Compare the new run against the\n*previous* 4.x run (B-new vs B-prior), not only vs 3.x. The diff should be\n*surgical* — e.g. the within-Latin letter gate moved exactly 6 files\n(IBM850 \u002F x-MacRoman → windows-1252) vs the prior run and nothing else. A\nbigger-than-expected diff means the change fired more broadly than intended.\n\n## Per-file detector attribution (`tk:encoding-detection-trace`)\n\nEvery JSON extract from a chain with multiple detectors carries\n`tk:encoding-detection-trace` in metadata. It's a per-detector emission\nlog with the META detector's arbitration tag at the end:\n\n```\nMojibusterEncodingDetector->Shift_JIS[STATISTICAL](1.00) [junk-filter-selected]\n```\n\nWhen investigating \"why did B pick X for this file?\", read this trace first\n— it tells you which base detector(s) emitted candidates and which one the\nmeta detector chose. If the trace shows ONLY Mojibuster firing with a CJK\npick, the bug is in Mojibuster's emission (pool too narrow), not in\nJunkFilter's arbitration.\n\n`tk:encoding-detector` is the simple-name credit string;\n`tk:detected-encoding` is the final answer (also in `Content-Encoding`).\n\n## Reproducing a single-file detection without a full chain\n\n```bash\n.\u002Fmvnw -q -pl tika-ml\u002Ftika-ml-junkdetect -Dmaven.repo.local=$(pwd)\u002F.local_m2_repo \\\n  -Dexec.classpathScope=test \\\n  -Dexec.mainClass=org.apache.tika.ml.junkdetect.TraceJunkFilter \\\n  -Dexec.args=\"--file \u003Cpath> --auto-candidates --content-cleaner --head-bytes 524288 --sample 120\" \\\n  exec:java\n```\n\nKey flags:\n- `--auto-candidates` — use Mojibuster's per-file pool as the candidate set\n- `--content-cleaner` — decode each candidate then run text through\n  `HtmlContentCleaner` to match the live chain\n- `--head-bytes 524288` — read up to 512 KB raw to match\n  `AdaptiveProbe.DEFAULT_RAW_CAP`. The default `READ_LIMIT` of 16 KB will\n  give a *different* probe than the live chain on long markup-heavy pages\n  and lead you to disagree with the live chain's pick. Always pass this\n  when reconciling a TraceJunkFilter run with a live extract.\n\nWithout `--head-bytes`, you are looking at a different probe than the\nchain saw — this is the most common source of \"trace says X, chain\nsays Y\" confusion.\n",{"name":5,"description":7},"main","9f8b48425fea3630020924f6ae96e98eabb9ce12",".skills\u002Fdevs\u002Ftika-eval-encoding-regression",1037,"https:\u002F\u002Fgithub.com\u002Fapache\u002Ftika","https:\u002F\u002Ftika.apache.org\u002F","https:\u002F\u002Fgithub.com\u002Fapache","https:\u002F\u002Fwww.apache.org\u002F","2026-11-13 03:03:34",{"source":8,"skillName":5,"commands":48,"prompt":58,"zipAvailable":59},[49,52,55],{"tool":50,"command":51},"npx","npx skills add apache\u002Ftika --skill tika-eval-encoding-regression",{"tool":53,"command":54},"bunx","bunx skills add apache\u002Ftika --skill tika-eval-encoding-regression",{"tool":56,"command":57},"pnpm","pnpm dlx skills add apache\u002Ftika --skill tika-eval-encoding-regression","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fapache\u002Ftika\u002Fblob\u002Fmain\u002F.skills\u002Fdevs\u002Ftika-eval-encoding-regression\u002FSKILL.md",true,[61,67,71,75,79,83],{"path":62,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":65,"sourceHtmlUrl":66},"evals\u002Feval.yaml","REFERENCE","参考文档",5985,"https:\u002F\u002Fgithub.com\u002FAzure\u002Fazure-sdk-for-net\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fazsdk-common-generate-sdk-locally\u002Fevals\u002Feval.yaml",{"path":68,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":69,"sourceHtmlUrl":70},"evals\u002Ftrigger.eval.yaml",1158,"https:\u002F\u002Fgithub.com\u002FAzure\u002Fazure-sdk-for-net\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fazsdk-common-generate-sdk-locally\u002Fevals\u002Ftrigger.eval.yaml",{"path":72,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":73,"sourceHtmlUrl":74},"fixtures\u002Ftspconfig.yaml",289,"https:\u002F\u002Fgithub.com\u002FAzure\u002Fazure-sdk-for-net\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fazsdk-common-generate-sdk-locally\u002Ffixtures\u002Ftspconfig.yaml",{"path":76,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":77,"sourceHtmlUrl":78},"references\u002Fcustomization-workflow.md",6747,"https:\u002F\u002Fgithub.com\u002FAzure\u002Fazure-sdk-for-net\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fazsdk-common-generate-sdk-locally\u002Freferences\u002Fcustomization-workflow.md",{"path":80,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":81,"sourceHtmlUrl":82},"references\u002Fdetailed-workflow.md",1698,"https:\u002F\u002Fgithub.com\u002FAzure\u002Fazure-sdk-for-net\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fazsdk-common-generate-sdk-locally\u002Freferences\u002Fdetailed-workflow.md",{"path":84,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":85,"sourceHtmlUrl":86},"references\u002Fsdk-repos.md",3194,"https:\u002F\u002Fgithub.com\u002FAzure\u002Fazure-sdk-for-net\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fazsdk-common-generate-sdk-locally\u002Freferences\u002Fsdk-repos.md",[88,128,154,180,206,229],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":105,"publishedAt":33},"01M6ACFKZR081Z7QX5XB1ZA2R2","file-to-markdown","Turn almost any file into Markdown plus metadata — PDF, Office, HTML, email, archives, images, audio\u002Fvideo, 1000+ formats — powered by Apache Tika, either via the tika-app CLI (zero setup, one file) or a running tika-server (curl, warm process, many calls). Leads with rmeta (structured, embedded-item-aware output) as the default operation rather than flat concatenated text, since you can't tell whether a file has embedded content from its extension. Covers metadata-only triage, type and language detection, OCR, and output-size discipline for agent context. Use whenever a task involves reading the content of a file whose format you don't want to hand-parse.","https:\u002F\u002Fgithub.com\u002Fapache\u002Ftika\u002Fblob\u002Fmain\u002F.skills\u002Fusers\u002Ffile-to-markdown\u002FSKILL.md",[94,99,102],{"categoryNo":95,"categoryName":96,"taxonomyType":97,"taxonomyTypeDesc":98},"01JBSEEDCAT000000000000002","文档处理","SCENARIO","使用场景",{"categoryNo":100,"categoryName":101,"taxonomyType":97,"taxonomyTypeDesc":98},"01JBSEEDCAT000000000000016","图像与多媒体",{"categoryNo":103,"categoryName":104,"taxonomyType":97,"taxonomyTypeDesc":98},"01JBSEEDCAT000000000000023","Skill与智能体开发",[106,24,107,108,109,110,111,9,112,113,19,114,115,116,117,118,119,120,121,122,123,124,125,126,127],"file to markdown","markdown","PDF","cli","triage","setup","content","metadata","server","ocr","Video","Audio","text","discipline","only","images","default","either","app","you","hand","parse",{"skillNo":129,"name":130,"nameZh":6,"description":131,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":132,"categories":133,"keywords":138,"publishedAt":33},"01M6ACFHJR3612ZE6RNX1P5WG9","file-to-markdown-docker","Run Apache Tika as a Docker container when you need guaranteed OCR (scanned PDFs, images) or geospatial raster support with zero local install — `apache\u002Ftika:\u003Cversion>-full` bundles Tesseract, GDAL, ImageMagick, and fonts. Also covers the minimal image, port\u002Fvolume\u002Fmemory conventions, the path-identity mount gotcha, and how to confirm OCR actually ran rather than silently returning no text. Powered by Apache Tika. Use when a local `tika-app`\u002F`tika-server` doesn't have Tesseract installed, or you want a disposable, self-contained parsing environment. Companion to the `file-to-markdown` skill, which covers the parsing calls themselves once a server is up.","https:\u002F\u002Fgithub.com\u002Fapache\u002Ftika\u002Fblob\u002Fmain\u002F.skills\u002Fusers\u002Ffile-to-markdown-docker\u002FSKILL.md",[134,135],{"categoryNo":95,"categoryName":96,"taxonomyType":97,"taxonomyTypeDesc":98},{"categoryNo":136,"categoryName":137,"taxonomyType":97,"taxonomyTypeDesc":98},"01JBSEEDCAT000000000000011","运维与部署",[139,140,106,24,107,141,142,143,144,145,9,19,115,118,146,121,147,148,149,124,150,125,151,152,153],"docker","file to markdown docker","image","memory","support","identity","run","up","self","volume","local","how to","fonts","which","when",{"skillNo":155,"name":156,"nameZh":6,"description":157,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":158,"categories":159,"keywords":161,"publishedAt":33},"01M6ACFE1684DE184VE0D7PQVY","file-forensics","Examine what a file claims about itself and what it actually contains — powered by Apache Tika. True content-based type detection (extensions lie), provenance claims (authors, dates, creating application), revision and tamper signals (PDF incremental updates, tracked changes, hidden slides, zip integrity), hidden and embedded content (attachments, macros), risk indicators (PDF JavaScript actions, encryption), and content digests. Evidence gathering, not verdicts: Tika reports what the file asserts and what parsing observed; it does not attribute authorship or validate signatures. Use for triaging suspicious files, provenance questions, e-discovery-style review, or \"is this file what it claims to be.\"","https:\u002F\u002Fgithub.com\u002Fapache\u002Ftika\u002Fblob\u002Fmain\u002F.skills\u002Fusers\u002Ffile-forensics\u002FSKILL.md",[160],{"categoryNo":95,"categoryName":96,"taxonomyType":97,"taxonomyTypeDesc":98},[162,24,163,108,164,165,166,167,9,112,19,168,169,170,171,172,173,174,175,30,176,177,178,179],"file forensics","forensics","review","suspicious","javascript","validate","actions","encryption","risk","integrity","evidence","reports","files","signatures","revision","zip","updates","claims",{"skillNo":181,"name":182,"nameZh":6,"description":183,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":184,"categories":185,"keywords":186,"publishedAt":33},"01M6ACFB92SPADF0FY4490B847","update-site-for-release","Update\u002Fpublish the Apache Tika website (tika-site SVN repo) for a release — step 17 of the Release Process. Handles the 4.x track (Changes page + aggregate javadoc + Antora docs branch) vs the 3.x maintenance track (full per-version apt docs + javadoc). Use for \"update the site\", \"publish the site for X.Y.Z\", \"the website part of the release\".","https:\u002F\u002Fgithub.com\u002Fapache\u002Ftika\u002Fblob\u002Fmain\u002F.skills\u002Fdevs\u002Fupdate-site-for-release\u002FSKILL.md",[],[187,188,189,190,32,191,192,9,19,193,194,195,196,197,198,199,200,201,202,203,204,205],"update","release","update site for release","site","docs","release process","website","apt","publish","page","branch","repo","use","version","maintenance","full","svn","changes","track",{"skillNo":207,"name":208,"nameZh":6,"description":209,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":210,"categories":211,"keywords":218,"publishedAt":33},"01M6ACF9G52NYHXQ0BVSJ3VH6E","tika-eval-h2-query","Query the tika-eval H2 database directly for counts and joins the canned reports do not compute — connection gotchas, key tables, example queries. Use when the xlsx\u002Fsummary.md reports are not enough.","https:\u002F\u002Fgithub.com\u002Fapache\u002Ftika\u002Fblob\u002Fmain\u002F.skills\u002Fdevs\u002Ftika-eval-h2-query\u002FSKILL.md",[212,215],{"categoryNo":213,"categoryName":214,"taxonomyType":97,"taxonomyTypeDesc":98},"01JBSEEDCAT000000000000004","电子表格",{"categoryNo":216,"categoryName":217,"taxonomyType":97,"taxonomyTypeDesc":98},"01JBSEEDCAT000000000000010","后端与接口",[19,219,23,220,221,222,199,223,173,27,224,225,226,227,228,32,153],"tika eval h2 query","query","xlsx","database","example","do","summary","compute","connection","tables",{"skillNo":230,"name":231,"nameZh":6,"description":232,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":233,"categories":234,"keywords":235,"publishedAt":33},"01M6ACF53J5MJCDNAEMCBV0RPP","tika-eval-compare","Compare extracts from two Tika builds over a corpus to detect regressions in content, encoding, exceptions, and embedded-document handling. Use for \"compare before\u002Fafter extracts\", \"eval this change against the corpus\".","https:\u002F\u002Fgithub.com\u002Fapache\u002Ftika\u002Fblob\u002Fmain\u002F.skills\u002Fdevs\u002Ftika-eval-compare\u002FSKILL.md",[],[19,236,23,237,238,112,21,199,239,240,241,32],"tika eval compare","compare","change","embedded","document","detect"]