tika-eval-compare

<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->

Local override: $TIKA_SKILLS_LOCAL/tika-eval-compare/LOCAL.md (default ~/.tika-skills), read after this file, wins on conflict.

tika-eval: Compare Before/After Extracts

Compare the output of two versions of Tika against a corpus of files to detect regressions in content extraction, encoding, exceptions, and embedded document handling.

Before You Start

Ask the user for:

1. Working directory — where to put builds, extracts, eval db, and reports (<workdir> below). All artifacts go here.
2. Number of threads (-n) — default is 2. Use -n 6 for faster runs when parse time comparison is not needed. When comparing parse times between A and B, use the same -n for both.
3. Run reports? — whether to auto-generate the HTML/Excel reports and summary.md at the end (the -r flag on tika-eval Compare).

Prerequisites

• Two tika-app builds (a "before" and an "after"), each as an unzipped zip archive containing tika-app-*.jar, lib/, and plugins/.
• A corpus of input files (a directory tree).
• tika-eval-app, built from tika-eval/tika-eval-app (use the zip). The bare target/tika-eval-app-*.jar is thin (no bundled deps) and dies with NoClassDefFoundError: ...GzipCompressorOutputStream (esp. with -r). Always run the jar from the unzipped target/*.zip dir, which carries its lib/.
• Enable MD5 digesting in both configs so tika-eval can match embedded documents by content hash (not just index position). Add to the config JSON:

  "parse-context": {
    "commons-digester-factory": {
      "digests": [
        { "algorithm": "MD5" }
      ]
    }
  }
  

  Note: parse-context is a JSON object, not an array.

Step 1 — Generate Extracts

Run each tika-app version against the same input corpus. The batch mode is triggered automatically when the first positional argument is a directory:

java -jar <before>/tika-app-*.jar <input-dir> <extracts-a-dir>
java -jar <after>/tika-app-*.jar  <input-dir> <extracts-b-dir>

To use a custom config (e.g., SAX vs DOM parsers, deleted content, macros), pass --config=<file.json> before the input/output dirs:

java -jar <tika-app>/tika-app-*.jar --config=dom-config.json <input-dir> <extracts-a-dir>
java -jar <tika-app>/tika-app-*.jar --config=sax-config.json <input-dir> <extracts-b-dir>

Each run walks the input directory recursively and writes one .json file per input file (recursive metadata + XHTML content, equivalent to tika-app -J). The directory structure mirrors the input.

Provenance + crash ledger (preferred)

run-batch.sh (next to this file) wraps the same invocation and records what ran, so an extract set can be tied to a build after the fact and a crashed file is distinguishable from one that parsed to nothing:

.skills/devs/tika-eval-compare/run-batch.sh --app <before> --input <input-dir> --extracts <extracts-a-dir> --note baseline
.skills/devs/tika-eval-compare/run-batch.sh --app <after>  --input <input-dir> --extracts <extracts-b-dir> --note candidate [--config cfg.json]

It writes run-info-<run.id>.json and, when the app's file-system plugin ships the jsonl reporter (TIKA-4846), crashes-<run.id>.jsonl into <extracts>/.run-info/; Compare picks them up from there by default (-ra/-rb, -pa/-pb override). A baseline without the reporter gets run-info but no ledger. Needs python3.

Launching tika-app directly (a standing run script with its own config, as on a corpus box) skips run-batch.sh, so add the reporter to that config once with the ledger path read from the environment, and export TIKA_EXTRACTS=<extracts-dir> in the same shell as the run. An unset variable fails the load naming it and the JSON path, so a run cannot silently proceed without its ledger. The file name must keep the crashes- prefix for Compare/Profile to find it by default:

"pipes-reporters": {
  "file-system-jsonl-reporter": {
    "path": "${env:TIKA_EXTRACTS}/.run-info/crashes-batch.jsonl",
    "includes": ["OOM", "TIMEOUT", "UNSPECIFIED_CRASH", "FAILED_TO_INITIALIZE",
                 "FETCHER_INITIALIZATION_EXCEPTION", "EMITTER_INITIALIZATION_EXCEPTION",
                 "CLIENT_UNAVAILABLE_WITHIN_MS", "FETCH_EXCEPTION", "EMIT_EXCEPTION",
                 "FETCHER_NOT_FOUND", "EMITTER_NOT_FOUND", "PAYLOAD_LIMIT_EXCEEDED"],
    "onExists": "EXCEPTION",
    "maxMessageLength": 4096
  }
}

Notes

• Do NOT pass -n <N> as a trailing argument — it confuses the async mode auto-detection. If you need to control parallelism, use -i <input-dir> -o <output-dir> -n <N> with explicit flags.
• Default parallelism is 2 forked JVM clients. Use -n 6 for faster runs when parse time comparison is not needed. When comparing parse times, keep -n the same for both A and B.
• Default timeout is 30 000 ms per file.

Step 2 — Run tika-eval Compare

Unzip the tika-eval-app zip, then run:

java -jar <tika-eval>/tika-eval-app-*.jar Compare \
  -a <extracts-a-dir> \
  -b <extracts-b-dir> \
  -d <db-path> \
  -r \
  -rd <reports-dir>

Flag	Description
-a	Directory of "before" extracts (required)
-b	Directory of "after" extracts (required)
-d	H2 database path (temp file if omitted)
-r	Auto-run Report + tgz the reports dir (<reportsDir>.tgz) after Compare
-rd	Reports output directory (default: reports)
-z	Gzip the H2 db (<db>.mv.db.gz) after Compare for transfer; requires -d (no-op + warning for a temp db). Combine with -r to package both.
-n	Number of worker threads
-pa/-pb	jsonl ledger for A/B (default: <extracts>/.run-info/crashes-*.jsonl); fills containers.pipes_status_a/b
-ra/-rb	run-info json for A/B (default: <extracts>/.run-info/run-info-*.json); lands in run_info_a/b. Refused unless the matching ledger is named crashes-<run.id>.jsonl; discovery refuses a .run-info holding more than one run-info or ledger

With -pa/-pb, summary.md and exceptions/extract_exceptions_by_pipes_status_*.xlsx split NO_EXTRACT_FILE into CRASH (OOM/TIMEOUT/UNSPECIFIED_CRASH), any other recorded status as-is, NO_PIPES_RECORD (ledger has no line: the batch recorded no failure), BATCH_WITHOUT_LEDGER, and NO_PIPES_REPORT_SUPPLIED. A crash status with an extract present is a success whose status was lost — listed separately, not a failure. run_info_a/b.pipes_report.joined says how many containers matched a ledger row; zero with a non-empty ledger means the wrong ledger, or a crawl without -i (which never sees files that crashed).

Step 3 — Review Results

Reports are written as Excel .xlsx files under the reports directory, plus a summary.md with key metrics:

• Content Quality (Dice Coefficient) — similarity between A and B per mime type. Mean dice < 0.95 warrants investigation.
• OOV / Languageness Changes — increased out-of-vocabulary rate or decreased languageness z-score may indicate encoding regressions.
• Content Length Ratio Outliers — files where B is >2× or <0.5× the length of A.
• Exception Changes — new exceptions in B or fixed exceptions.
• Embedded Document Count Changes — gained/lost attachments.
• Content Regressions — lowest-dice files with token-level diffs.
• Content Lost / Gained — files that went empty↔non-empty.

Interpreting Results

Metric	Good	Investigate
Mean dice (same mime)	≥ 0.95	< 0.90
New exceptions in B	0	> 0 — every one needs explanation
Embedded doc count losses	0	> 0 — investigate by mime type
OOV delta	< 0.05	> 0.10
Content length ratio	0.5–2.0	> 5× or < 0.2×
Exception count	≤ A	> A
Total files (B) vs (A)	equal or higher	lower — missing embedded docs

Encoding-detection evals

For charset/encoding-detector changes, the summary reports don't cover it — query the db directly (see the tika-eval-h2-query skill). The detected encoding is in the ENCODINGS_A/ENCODINGS_B tables (DETECTED_ENCODING, ENCODING_DETECTOR, DECLARED_METADATA), not PROFILES. Key signals: per-encoding counts (e.g. CJK total), A→B flips by direction, and OOV on the flipped files (a flip that worsens OOV is a regression; one that improves it is a fix). Pair on ID; map back to the source file via PROFILES_*.FILE_NAME (the content hash).

CRITICAL: Review Checklist

The purpose of tika-eval is to find regressions BEFORE a release. After reading summary.md, report each of these to the user:

1. New exceptions: Exact count. If > 0, investigate stack traces. Every new exception is a bug.
2. Total files delta: "Total files (A)" vs "(B)". If B < A, embedded documents are being lost — aggregate the losses by child mime type.
3. Embedded doc count changes: Report losers from the summary table.
4. Dice scores: Flag any mean < 0.99 for OOXML or < 0.95 for others.
5. Content length outliers: Flag ratio > 3x or < 0.3x.
6. Fixed exceptions: Report the count.

Do not summarize results as "looks good" based on dice scores alone. Dice measures text similarity, not attachment completeness. Always check the Total files delta.

After fixing regressions, re-run the full eval. A fix for one format may not cover another. Verify the numbers moved and no new issues appeared.

Building from Source

# tika-app
./mvnw clean install -pl tika-app -am -Pfast \
  -Dmaven.repo.local=$(pwd)/.local_m2_repo

# tika-eval-app
./mvnw clean install -pl tika-eval/tika-eval-app -am -Pfast \
  -Dmaven.repo.local=$(pwd)/.local_m2_repo

The zip artifacts are in <module>/target/<module>-*.zip.

Example: Comparing MSG parsing changes

# Download the "before" snapshot
curl -o /tmp/tika-app-before.zip <snapshot-url>
unzip -qo /tmp/tika-app-before.zip -d /tmp/tika-app-before

# Build the "after" from local changes
./mvnw clean install -pl tika-app -am -Pfast \
  -Dmaven.repo.local=$(pwd)/.local_m2_repo
unzip -qo tika-app/target/tika-app-*.zip -d /tmp/tika-app-after

# Generate extracts
java -jar /tmp/tika-app-before/tika-app-*.jar <corpus> /tmp/extracts-a
java -jar /tmp/tika-app-after/tika-app-*.jar  <corpus> /tmp/extracts-b

# Build and run tika-eval
./mvnw clean install -pl tika-eval/tika-eval-app -am -Pfast \
  -Dmaven.repo.local=$(pwd)/.local_m2_repo
unzip -qo tika-eval/tika-eval-app/target/tika-eval-app-*.zip -d /tmp/tika-eval

java -jar /tmp/tika-eval/tika-eval-app-*.jar Compare \
  -a /tmp/extracts-a -b /tmp/extracts-b \
  -d /tmp/eval-db -r -rd /tmp/eval-reports

# Review
cat /tmp/eval-reports/summary.md