tao-analyze-gaps-vlm-bcq
Extract false-positive and false-negative gaps from VLM binary-classification-question (BCQ, yes/no) predictions. Use when the user asks to "analyze VLM BCQ gaps", "extract VLM false positives and false negatives", or identify failure cases from a predictions JSON for DEFT root-cause analysis on a binary-classification VLM workflow.
- License 许可
- Apache-2.0
- 最近更新
- 2026-07-18
- Stars
- ★ 0
- Forks
- 1
01 前言 Front Matter
- name
- tao-analyze-gaps-vlm-bcq
- tags
- [ "gap-analysis", "rcca", "vlm", "evaluation", "false-positive", "false-negative" ]
- license
- Apache-2.0
- metadata
- { "author": "NVIDIA Corporation", "version": "0.1.0" }
- description
- Extract false-positive and false-negative gaps from VLM binary-classification-question (BCQ, yes/no) predictions. Use when the user asks to "analyze VLM BCQ gaps", "extract VLM false positives and false negatives", or identify failure cases from a predictions JSON for DEFT root-cause analysis on a binary-classification VLM workflow.
- allowed-tools
- Read Bash
- compatibility
- Requires docker + nvidia-container-toolkit.
03 SKILL.md 正文
VLM Binary Classification Gap Analysis
Reads a VLM predictions JSON, compares each model response against ground truth, and writes FP/FN failure cases to a JSONL file with a summary report.
Purpose
After running a VLM on a binary yes/no evaluation task, the predictions need to be compared against ground truth to identify failure cases. This skill produces a structured list of FP (false positive) and FN (false negative) samples that downstream RCCA stages (e.g., cosmos generation, root cause analysis) consume to drive a DEFT iteration.
Usage
Invoke the vlm_bcq action inside the TAO Toolkit data services container with Hydra-style key=value overrides:
gap_analysis vlm_bcq \
predictions_json=/path/to/results.json \
results_dir=/path/to/output/gaps
Include videos_dir when video_id values in the predictions are relative paths:
gap_analysis vlm_bcq \
predictions_json=/path/to/results.json \
results_dir=/path/to/output/gaps \
videos_dir=/path/to/videos/root
After the run, surface the FP/FN counts from kpi_gaps_report.txt and point downstream stages at kpi_gaps.jsonl.
Inputs
- predictions_json: Path to predictions JSON file. Must be a JSON array where each item has
video_id,response, andgtfields.responseandgtare parsed with word-boundary matching —'yes'or'no'anywhere in the string is recognized. Samples where both or neither are present are skipped with a warning. - videos_dir (optional): Base directory for resolving relative
video_idpaths. If omitted,video_idvalues are used as absolute paths.
Predictions JSON format:
[
{
"video_id": "/path/to/video.mp4",
"response": "Yes, there is a collision.",
"gt": "B. No",
"question": "Is there a collision?"
}
]
Outputs
- kpi_gaps.jsonl: One JSON object per line for each FP/FN case. Fields:
video_id(absolute path),error_type(FPorFN),question,ground_truth,response. - kpi_gaps_report.txt: Human-readable table with total FP/FN counts.
If no gaps are found, no files are written and a message is logged.
Key Parameters
| Parameter | Required | Description |
|---|---|---|
| predictions_json | Yes | Path to predictions JSON file |
| results_dir | Yes | Output directory; created if it does not exist |
| videos_dir | No | Base directory for resolving relative video_id paths |
Error Patterns
| Error | Cause | Fix |
|---|---|---|
FileNotFoundError |
predictions_json does not exist |
Check the path |
ValueError: must be a JSON array |
Predictions file is not a list | Wrap predictions in [...] |
ValueError: missing 'gt'/'response'/'video_id' |
A prediction item is missing a required field | Inspect and fix the predictions JSON |
| Samples silently skipped | response or gt contains both or neither 'yes'/'no' |
Check logs for warnings; inspect those samples |
04 同仓库更多 Skills
arxiv-search
Search arXiv physics, math, and computer science preprints using natural language queries. Powered by Valyu semantic search.
arxiv-search
Search arXiv physics, math, and computer science preprints using natural language queries. Powered by Valyu semantic search.
working-with-legacy-code
Safely change and test untested codebases using Feathers' "Working Effectively with Legacy Code". Use when the user mentions "legacy code", "no tests", "untested codebase", "how do I test this", "seams", "characterization tests", "golden master", "sprout method", "afraid to change this code", "monster method", "dependency breaking", or "inherited a messy codebase". Also trigger when changing code without tests safely, getting a class under test when constructors, statics, or singletons block it, adding features to tangled modules, or planning incremental test coverage for an old codebase. Covers the legacy-code change algorithm, seams, characterization tests, sprout/wrap, and dependency-breaking techniques. For refactoring code that already has tests, see refactoring-patterns. For day-to-day code quality, see clean-code.
working-with-legacy-code
Safely change and test untested codebases using Feathers' "Working Effectively with Legacy Code". Use when the user mentions "legacy code", "no tests", "untested codebase", "how do I test this", "seams", "characterization tests", "golden master", "sprout method", "afraid to change this code", "monster method", "dependency breaking", or "inherited a messy codebase". Also trigger when changing code without tests safely, getting a class under test when constructors, statics, or singletons block it, adding features to tangled modules, or planning incremental test coverage for an old codebase. Covers the legacy-code change algorithm, seams, characterization tests, sprout/wrap, and dependency-breaking techniques. For refactoring code that already has tests, see refactoring-patterns. For day-to-day code quality, see clean-code.
web-typography
Select, pair, and implement typefaces for web projects. Use when the user mentions "font pairing", "which typeface", "line height", "responsive typography", "web font loading", "type hierarchy", "variable fonts", "FOUT/FOIT", "typographic scale", or "the text is hard to read". Also trigger when choosing between system fonts and web fonts, optimizing font-loading performance, or designing readable long-form content. Covers readability evaluation, CSS implementation, and performance optimization. For overall UI design systems, see refactoring-ui. For dramatic typographic experiences, see top-design.
web-typography
Select, pair, and implement typefaces for web projects. Use when the user mentions "font pairing", "which typeface", "line height", "responsive typography", "web font loading", "type hierarchy", "variable fonts", "FOUT/FOIT", "typographic scale", or "the text is hard to read". Also trigger when choosing between system fonts and web fonts, optimizing font-loading performance, or designing readable long-form content. Covers readability evaluation, CSS implementation, and performance optimization. For overall UI design systems, see refactoring-ui. For dramatic typographic experiences, see top-design.