tao-generate-image-grounding
Two-step image grounding pipeline: extracts referring expressions from (image, caption) pairs and grounds them to pixel-space bounding boxes via a VLM. Use when the user wants to ground captions to bboxes, generate phrase-grounded annotations, auto-label images for grounding, or run the image_grounding pipeline. Triggers include 'image grounding', 'phrase grounding', 'ground captions', 'auto-label image grounding', 'image_grounding'.
- License 许可
- Apache-2.0
- 最近更新
- 2026-10-08
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- tao-generate-image-grounding
- tags
- [ "image", "grounding", "bounding-boxes", "auto-label", "vlm", "2d-grounding" ]
- license
- Apache-2.0
- metadata
- { "author": "NVIDIA Corporation", "version": "0.1.0" }
- description
- Two-step image grounding pipeline: extracts referring expressions from (image, caption) pairs and grounds them to pixel-space bounding boxes via a VLM. Use when the user wants to ground captions to bboxes, generate phrase-grounded annotations, auto-label images for grounding, or run the image_grounding pipeline. Triggers include 'image grounding', 'phrase grounding', 'ground captions', 'auto-label image grounding', 'image_grounding'.
- allowed-tools
- Read Bash Write
- compatibility
- Requires docker + nvidia-container-toolkit + at least one VLM endpoint (Gemini API key or OpenAI-compatible).
02 资源文件 7
- 参考文档BENCHMARK.md3.8 KB
- 参考文档metadata.json675 B
- 参考文档references/configuration.md9.3 KB
- 参考文档references/skill_info.yaml1015 B
- 参考文档references/vllm_server.md6.7 KB
- 参考文档skill-card.md3.8 KB
- 参考文档verification.json2.1 KB
03 SKILL.md 正文
Image Grounding Pipeline
Turn (image, caption) pairs into per-image grounded annotations: cleaned captions, referring expressions with character spans, and pixel-space bounding boxes for each expression. A single VLM (Gemini or any OpenAI-compatible endpoint) handles both steps.
Purpose
Generate phrase-grounded training data for referring-expression and grounding models. The VLM acts as a "teacher" annotator: Step 0 extracts referring expressions from the caption while looking at the image; Step 1 returns one bbox set per expression for each image.
Pipeline Architecture
Step 0: Expression extraction → VLM cleans caption, extracts referring expressions + char spans
Step 1: Phrase grounding → VLM returns pixel bboxes + scores per expression
Steps are individually selectable via workflow.steps. Each step writes a per-sample checkpoint to step_<N>_*/.ckpt/<sample_id>.json and skips already-processed records on re-run. Set workflow.force_reprocess: true to ignore checkpoints and reprocess from scratch.
Instructions
Initial setup
When a user wants to run this pipeline, walk through these steps:
-
Input JSONL: Ask for the JSONL path. Each line must be one object like
{"image_path": "...", "caption": "..."}.image_pathcan be absolute or relative. -
Image root: If any
image_pathvalues are relative, setdata.image_rootto the directory they should resolve from. -
API access: Ask the user which VLM endpoint they want to use. Present these five options and act on the choice:
- Gemini — set
vlm.backend: "gemini"; requireGOOGLE_API_KEY(env var orvlm.gemini.api_key). - NIM (e.g.
https://inference-api.nvidia.com/v1) — setvlm.backend: "openai"; collectbase_url,model_name, andapi_key. - TAO inference microservice (self-hosted, OpenAI-compatible). Confirm whether the server is already running:
- Running — collect
base_url,model_name, and (optionally)api_key; setvlm.backend: "openai". - Not running — guide the user through the
skills/applications/tao-run-inference-serviceskill, which stands up a local TAO inference microservice with an OpenAI-compatible API. Before promising a specific model, checkskills/applications/tao-run-inference-service/references/service.yamlforvalid_network_arch_config_basenames. Once the server is up, collectbase_url,model_name, and (optionally)api_key; setvlm.backend: "openai".
- Running — collect
- vLLM (self-hosted, OpenAI-compatible). Confirm whether the server is already running:
- Running — collect
base_url,model_name, and (optionally)api_key; setvlm.backend: "openai". - Not running — follow references/vllm_server.md to install and launch a vLLM server, then collect
base_url,model_name, and (optionally)api_key; setvlm.backend: "openai".
- Running — collect
- Custom (any other OpenAI-compatible endpoint) — set
vlm.backend: "openai"; collectbase_url,model_name, and (optionally)api_key.
If the user has no endpoint and does not want to set one up, stop and help resolve API access first.
- Gemini — set
-
Workflow steps: Choose one of:
- Full pipeline:
["0", "1"] - Expression extraction only:
["0"] - Grounding only:
["1"], which requires existing step-0 output atresults_dir/step_0_expression_extraction/annotations.jsonl
- Full pipeline:
-
Resume vs fresh run: By default, the workflow reuses checkpoints and skips completed records. To reprocess everything, set
image_grounding.workflow.force_reprocess=true.
Running the pipeline
The pipeline runs inside the TAO Toolkit container via the auto_label CLI:
auto_label generate -e /path/to/spec.yaml \
results_dir=/results \
image_grounding.data.input_jsonl=/data/captions.jsonl \
image_grounding.data.image_root=/data/images \
image_grounding.vlm.gemini.api_key=$GOOGLE_API_KEY
Generate a default spec: auto_label default_specs results_dir=/results module_name=auto_label, then set autolabel_type: "image_grounding". All fields support Hydra dot-notation overrides on the command line.
See references/configuration.md for the full YAML structure, all parameters, model/endpoint setup, and error patterns.
Recommended pilot workflow
- Run on 5-10 images with both steps
- Inspect
step_0_expression_extraction/annotations.jsonl— arecleaned_captionandexpressions[]accurate? Are the right noun phrases captured? - Inspect
step_1_grounding/annotations.jsonl— do the bboxes inexpressions[].instances[]look right? Are confidence scores reasonable? - If quality is insufficient, switch the VLM to a stronger model (e.g.
gemini-2.5-pro) or raisemedia_resolution/max_output_tokens, then re-run withforce_reprocess=true. - Scale to the full dataset once satisfied.
Configuration
Key configuration fields (full reference in references/configuration.md):
| Field | Default | Description |
|---|---|---|
workflow.steps |
["0","1"] |
Which pipeline steps to execute ("0" = expressions, "1" = grounding) |
workflow.max_workers |
4 |
Parallel threads per step (watch API rate limits) |
workflow.force_reprocess |
false |
Ignore per-sample checkpoints and reprocess from scratch |
vlm.backend |
"gemini" |
"gemini" or "openai" (OpenAI-compatible endpoint) |
data.input_jsonl |
required | Path to input JSONL with image_path + caption per line |
data.image_root |
"" |
Optional prefix for resolving relative image_path entries |
Inputs
A single JSONL file at data.input_jsonl. One JSON object per line:
| Field | Required | Description |
|---|---|---|
image_path |
yes | Absolute path, or relative path resolved against data.image_root |
caption |
yes | Free-text caption for the image |
image_id |
no | Stable identifier; auto-derived from the filename if missing |
width, height |
no | Image dimensions in pixels; default to 1920×1080 for bbox clamping if missing |
Outputs
All outputs go to results_dir/:
step_0_expression_extraction/annotations.jsonl— per-record output enriched withcleaned_captionandexpressions[](each withtext,expression_id,char_span,noun_chunk, emptyinstances[]).step_1_grounding/annotations.jsonl— same records withexpressions[].instances[]filled in (each instance hasbbox: [x1,y1,x2,y2]in pixel space,scorein[0.0, 1.0], andbbox_id).results_dir/annotations.jsonl— copy of the last step's output for convenience.step_<N>_*/.ckpt/<sample_id>.json— per-sample checkpoints used for resume.
Prerequisites
- Container:
nvcr.io/nvidia/tao/tao-toolkit:6.26.3-pyt - API access: At least one VLM endpoint (Gemini API key or OpenAI-compatible endpoint capable of image input)
04 同仓库更多 Skills
zkrouter
Self-hosted OpenAI-compatible LLM router — smart tier-based routing, ZK-stamped routing decisions, mode overrides, free OpenRouter at install via the Birth bot. Save 60-80% on inference costs by routing to the cheapest capable model across 12+ OpenRouter models.
zk
For custom ZK Solana programs and privacy-preserving applications to prevent double spending. Guide to integrate rent-free nullifier PDAs for double-spend prevention.
zk-omni-messaging
Zero-knowledge omnichain messaging between Robinhood Chain and Solana using CheshireZkOmniMessenger (msgType 4) with nullifier anti-replay and the zk-omni-relayer service.
zerion-wallet-data
Use when the user asks to check a wallet's portfolio, token balances, DeFi positions, PnL, transaction history, or NFTs. Provides interpreted on-chain data across 40+ EVM chains and Solana via x402 pay-per-request or API key — no raw RPC decoding needed.
youtube-clipper
YouTube 视频智能剪辑工具。下载视频和字幕,AI 分析生成精细章节(几分钟级别), 用户选择片段后自动剪辑、翻译字幕为中英双语、烧录字幕到视频,并生成总结文案。 使用场景:当用户需要剪辑 YouTube 视频、生成短视频片段、制作双语字幕版本时。 关键词:视频剪辑、YouTube、字幕翻译、双语字幕、视频下载、clip video
yield-optimization
Multi-chain yield optimization via yield.xyz (StakeKit). Discover 2,988+ yield opportunities across 75+ blockchains, build deposit/withdrawal transactions with shell scripts, and sign them with a MoonPay wallet. Use when the user wants to earn yield, find the best lending or staking rate, enter/exit a position, or check portfolio balances.