AGENT SKILL

parser-selection-methodology

Parser bake-off and selection — run tests against representative corpus, score fidelity and multimodal handling, measure cost and latency, produce comparison report with justified recommendation

License 许可
Apache-2.0
所属仓库
cwik-tech/octopus
最近更新
2026-10-27
Stars
★ 0
Forks
0

01 前言 Front Matter

name
parser-selection-methodology
description
Parser bake-off and selection — run tests against representative corpus, score fidelity and multimodal handling, measure cost and latency, produce comparison report with justified recommendation

03 SKILL.md 正文

Parser Selection Methodology

Purpose

Select parsers based on measured fidelity against your actual corpus, not vendor claims. Run a bake-off, score each candidate on structure preservation, multimodal quality, and operational metrics, and produce a comparison report that justifies the selection.

When to Use

  • Before committing to a parser for the production pipeline
  • After a parser update to check for regressions
  • When evaluating a new vendor or tool as a potential replacement

Methodology

1. Build a Representative Test Set

Sample your corpus across:

  • File types: PDF (native, scanned), DOCX, XLSX, images, drawings—everything you'll parse
  • Content complexity: Simple prose, complex nested structures, multi-column layouts, intricate tables
  • Languages: Every language in your corpus
  • Edge cases: Low-quality scans, diagrams without captions, tables with merged cells, multilingual mixed content

Target size: 50–200 documents. Represent 80%+ of your content types and at least 5–10 examples of your highest-risk categories.

2. Define the Evaluation Scorecard

For each parser, measure:

Structure Fidelity (40% weight)

  • Heading hierarchy preserved (H1 → H2 → H3)
  • Lists and bullets correctly identified and nested
  • Footnotes, references, hyperlinks extracted
  • Section boundaries clear in Markdown output
  • No spurious line breaks or text reordering

Multimodal Handling (30% weight)

  • Tables: cell content, merged cells, multi-row headers extracted correctly
  • Images: detected, described, or referenced in output
  • Diagrams and drawings: native extraction vs fallback description quality
  • Scanned pages: OCR character accuracy, layout preservation
  • OCR confidence scores available for all text

Operational Metrics (20% weight)

  • Latency per document (seconds)
  • Cost per document (currency)
  • Asset handling (images retained, referenced, or discarded)
  • Failure modes (formats rejected gracefully vs crashes)
  • Reproducibility (same input → same output)

Auditability (10% weight)

  • Provenance metadata available (source file, revision, parse timestamp)
  • Confidence scores per extracted component
  • Position information (page, byte offset, location) retained

3. Run the Bake-Off

For each parser candidate:

  1. Run against the full test set
  2. For each document, score each criterion against the scorecard
  3. Categorize failures: structure loss, multimodal quality drop, missing content, unsupported format, error/crash
  4. Aggregate scores by content type (tables, scans, diagrams, multilingual, etc.)
  5. Calculate cost and latency distributions (mean, p95, p99)
  6. Measure end-to-end pipeline time (ingestion + parsing + validation)

4. Build the Comparison Report

Format: Side-by-side matrix with weightings and totals.

Example:

Criterion Weight Parser A Parser B Parser C Winner
Heading hierarchy 10% 96% 94% 98% C
Table correctness 15% 92% 88% 95% C
OCR accuracy 10% 88% 91% 90% B
Diagram extraction 5% Native OCR Fallback A
Cost per doc 10% $0.03 $0.01 $0.02 B
Latency (p95 sec) 5% 2.1 1.8 2.3 B
Asset handling 10% Retained Reference Retained Tie
Confidence scores 10% Per-component Per-doc None A
Error handling 5% Graceful Crash Retry A
Weighted total 100% 90.2% 88.5% 92.1% C

Include:

  • Failure analysis: Which content types failed for each parser, and why
  • Cost-benefit: Total parsing cost at scale (documents × cost per doc) vs quality gain
  • Risk assessment: What happens if this parser becomes unsupported, or confidence degrades
  • Sensitivity analysis: How much does cost dominate vs fidelity for your use case

5. Recommendation

State the winner and your confidence:

  • Primary recommendation: [Parser name], because [specific strengths on your corpus: e.g., "95% table accuracy on XLSX, native diagram extraction, $0.01/doc"]
  • Fallback (if primary unavailable): [Parser name], score [X]%
  • Not recommended: [Parser name], because [specific weakness that matters: e.g., "92% OCR accuracy insufficient for scanned technical docs in your corpus"]

Acceptance criteria: Include the conditions under which this recommendation holds. Example: "This holds if OCR confidence stays >90% on scanned pages. If baselines degrade, we re-evaluate in Q3 2026."

Questions to Answer Before Running Bake-Off

  1. What is your test set size and composition, and does it represent your hardest content types?
  2. What is an acceptable OCR confidence floor for your use case? (e.g., 90%, 95%)
  3. For tables: merged cells, multi-row headers, complex layouts—what counts as correct?
  4. For scanned pages: is your goal character accuracy, layout preservation, or both?
  5. For diagrams: extract native vector data, generate text descriptions, or retain image?
  6. What is your cost budget per document, and at what volume does that budget matter?
  7. If a parser fails on 5% of your corpus, can you tolerate a manual review queue, or must you re-parse?
  8. Do you need incremental sync or only full corpus re-parsing?

Red Flags

  • A parser that scores well on the vendor's test set but poorly on your actual corpus
  • A parser that excels at PDFs but has no strategy for DOCX or scanned content
  • A vendor unwilling to provide confidence scores or failure categorization
  • Parser selection made on speed alone, without measuring fidelity on your corpus
  • A test set that excludes your highest-risk content types
  • Missing cost data or cost that scales non-linearly with document volume

Output

A parser comparison report containing:

  1. Test set description: Composition, size, sampling strategy, language distribution
  2. Evaluation scorecard: Criteria, weights, rationale for each weight
  3. Side-by-side comparison: Fidelity matrix with weighted totals
  4. Failure analysis: Failure rates by content type, root cause for each parser
  5. Cost-benefit analysis: Total cost at scale, cost per quality point gained
  6. Risk assessment: Parser stability, vendor support, contingency strategy
  7. Recommendation: Primary, fallback, not-recommended, with acceptance criteria
  8. Decision gate: Clear threshold for approval (e.g., "Primary recommendation approved if weighted score >90% and OCR confidence >90%")

04 同仓库更多 Skills

cwik-tech
cwik-tech/octopus

twenty-year-land-plan

Apply the Bren discipline of evaluating every land and development decision on a twenty-year horizon, with a comprehensive plan for how the land, the community, and the surrounding public infrastructure evolve together. Deals optimized for a five-year exit are structurally different from deals that compound across generations; insist on knowing which kind you are making. Use when evaluating land, entitlements, master-planned development, or phasing decisions, and when separating five-year exit deals from generational holds.

★ 0
cwik-tech
cwik-tech/octopus

thematic-gut-check

Apply the Gray discipline of identifying the structural theme behind every real estate deal, testing where that theme sits in its capital-flow lifecycle, and insisting that deals without a named theme — or deals late in a crowded theme — clear a higher bar. A mediocre deal in a strong theme usually beats a clever deal in a weak one. Use when naming the structural thesis behind a deal, judging whether that theme is early or crowded, and setting the bar the deal must clear.

★ 0
cwik-tech
cwik-tech/octopus

supply-pipeline-gut-check

Apply the Zell discipline of grounding every real estate thesis in the local supply pipeline. Demand counts over adjectives — units under construction, units permitted, units proposed — and convert them into a forward absorption schedule. A deal that ignores the pipeline is a bet on a market you have not actually studied. Use when validating a market thesis, forecasting absorption and rent growth, or counting units under construction, permitted, and proposed.

★ 0
cwik-tech
cwik-tech/octopus

ruin-scenario-mapping

Apply the Taleb discipline of identifying, for every real estate deal, the specific chain of events that produces ruin — permanent, unrecoverable capital loss — and verifying that no single such chain has non-trivial probability. Deals with any plausible ruin path must be restructured or abandoned. Use when sizing leverage, testing refinancing, covenant, and liquidity risk, or checking whether any single failure chain can permanently wipe out the position.

★ 0
cwik-tech
cwik-tech/octopus

revenue-cost-architecture

Design a venture's revenue streams, pricing logic, cost structure, and unit economics so the business model has a coherent theory of value capture, not just value creation. Use after the value proposition and customer segment are set and before financial projections, when choosing a pricing model, calculating contribution margin or CAC payback, or checking whether the unit economics hold at scale.

★ 0
cwik-tech
cwik-tech/octopus

replacement-cost-stress-test

Apply the Buffett discipline of pricing every real estate asset against the cost to rebuild it from scratch. The ratio of purchase price to replacement cost sets a ceiling on long-run downside: below replacement cost, new supply cannot profitably undercut you; at or above it, you are buying a liability disguised as an asset. Use when setting a purchase price, judging downside protection, or testing how exposed an asset is to newly built competing supply.

★ 0