AGENT SKILL

evaluation

Evaluate Unstructured partition outputs against gold outputs for text extraction, element types, table structure, and optional object detection metrics.

License 许可
Apache-2.0
最近更新
2026-09-03
Stars
★ 351
Forks
27

01 前言 Front Matter

name
evaluation
license
Apache 2.0
metadata
{ "disco-role": "operating" }
description
Evaluate Unstructured partition outputs against gold outputs for text extraction, element types, table structure, and optional object detection metrics.
disable-model-invocation
true

02 资源文件 3

03 SKILL.md 正文

Evaluation

Use this sub-skill when a user needs to compare existing Unstructured outputs with gold-standard outputs, interpret metric TSVs, or decide whether optional object-detection evaluation is appropriate.

Route Here For

  • Text extraction accuracy and missing-text checks between partition output JSON/TXT and gold TXT.
  • Element type frequency matching, including metadata.category_depth effects.
  • Table structure evaluation from Table elements, metadata.text_as_html, or metadata.table_as_cells.
  • Optional object detection metrics when torch and layout dump JSON files are available.
  • Safe, local comparison of two element JSON files before running broader metric batches.

Boundaries

  • To create or regenerate partition outputs, route to the partitioning sub-skill first.
  • To serialize, deserialize, or validate element JSON schemas, route to the elements-and-metadata sub-skill first.
  • Do not promise that a partition smoke test passed unless the user has run one in their own environment.
  • Treat object detection evaluation as optional because importing unstructured.metrics.evaluate can pull the torch-backed object detection module.

Quick Workflow

  1. Confirm that prediction and gold files are already generated and are paired by matching base filenames.
  2. Choose the metric family: text extraction, element type, table structure, table alignment/format conversion, or optional object detection.
  3. Validate file shape before batch evaluation: element JSON should be a list of element dictionaries; table JSON should include Table elements with table metadata or Deckerd-style cell data.
  4. For one pair of element JSON files, use scripts/evaluate_elements_pair.py for a dependency-light summary.
  5. For batch metrics, use the package calculators documented in references/metrics-reference.md.
  6. When a metric fails unexpectedly, check references/troubleshooting.md before changing the partitioning pipeline.

Pair Comparison Helper

python sub-skills/evaluation/scripts/evaluate_elements_pair.py \
  --prediction predicted.json \
  --gold gold.json \
  --table-source auto

The helper reports element count differences, text accuracy, missing text, element-type frequency match, and table structure metrics when compatible table data exists. It intentionally avoids importing unstructured.metrics.evaluate so ordinary text/element/table checks do not require torch.

Batch Calculator Pattern

For package-level TSV outputs, instantiate the metric calculator, pass prediction and gold directories, and write to an output directory outside the runtime skill tree:

from unstructured.metrics.evaluate import TextExtractionMetricsCalculator

TextExtractionMetricsCalculator("predictions", "gold", document_type="json").calculate(
    export_dir="metrics",
    visualize_progress=False,
    display_agg_df=False,
)

Use the detailed calculator table and output filenames in references/metrics-reference.md.

04 同仓库更多 Skills

VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-triton-opt

Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-solve

END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-skill

PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-pattern-fusion

Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-orchestrate

Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-feedback

Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).

★ 351