AGENT SKILL

vss-ask-video

Use this skill to ask the VSS agent's video_understanding tool a fresh visual question about a recorded clip. Not for prior tool output, search hits, or metadata-answerable questions.

License 许可
Apache-2.0
最近更新
2026-10-08
Stars
★ 0
Forks
0

01 前言 Front Matter

name
vss-ask-video
license
Apache-2.0
metadata
{ "tags": "nvidia blueprint operational", "version": "3.2.0", "github-url": "https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization" }
description
Use this skill to ask the VSS agent's video_understanding tool a fresh visual question about a recorded clip. Not for prior tool output, search hits, or metadata-answerable questions.

02 资源文件 5

03 SKILL.md 正文

Video QnA using VLM through VSS Agent

Use this skill when you need details about the video which requires VLM to look at the video frames — for example the agent has no usable prior answer and needs a fresh look at the pixels for a specific clip.


When to Use

  • The user asks what happens in the video, what objects / people / actions appear, colors, timing, safety, or other visual facts that require watching the clip.
  • The user asks for details that cannot be answered from existing messages, summaries, Elasticsearch/MCP results, or filenames alone—you need model inference on the video.
  • Follow-up questions about content details after a coarse summary or after report generation.

Do not use this skill when a database / MCP / prior tool output already answers the question, unless the user explicitly wants verification against the video.


Deployment prerequisite

This skill requires a VSS profile that serves the video_understanding tool — typically base (recommended) or lvs. Before any request:

  1. Probe the VSS agent:

    curl -sf --max-time 5 "http://${HOST_IP}:8000/docs" >/dev/null
    
  2. If the probe fails, ask the user:

    "No VSS profile is running on $HOST_IP. Shall I deploy base (recommended for per-clip VLM QnA) using the /vss-deploy-profile skill? If you prefer lvs, say so."

    • If yes → hand off to /vss-deploy-profile -p base (or -p lvs if the user prefers). Return here once it succeeds.
    • If no → stop.
  3. If the probe passes, proceed.


Sensor prerequisite

You MUST list VST sensors before any /generate call. This is required even when the user names the sensor explicitly, even when the user asserts the video is already uploaded, and even when a previous turn appeared to use the same video. Do not skip this step.

  1. List sensors:

    curl -sf --max-time 5 "http://${HOST_IP}:30888/vst/api/v1/sensor/list" | jq '.[].name'
    
  2. Compare the returned name values against the user-supplied <sensor-id> (or filename stem, e.g. warehouse_safety_0001).

  3. If a matching sensor is present → proceed to the Agent workflow below.

  4. If no matching sensor is present — upload the video first, then re-list to confirm the new sensor appears:

    # filename: must not contain whitespace
    # timestamp: ISO 8601 UTC — default 2025-01-01T00:00:00.000Z if user did not specify
    curl -s -X PUT "http://${HOST_IP}:30888/vst/api/v1/storage/file/<filename>?timestamp=<timestamp>" \
      -H "Content-Type: application/octet-stream" \
      -H "Content-Length: <file_size_in_bytes>" \
      --upload-file /path/to/<filename> | jq .
    

    See /vss-manage-video-io-storage for full upload semantics (v1 vs v2, conflict handling, delete flow). In interactive runs, confirm with the user before uploading. Never issue an unconditional PUT without first running the sensor-list check above — that is exactly the failure mode this prerequisite exists to prevent.


Agent workflow

The Sensor prerequisite above must have already confirmed (or made) the sensor exist on VST. Then:

  1. Clip — Identify sensor id, filename, or URL for one video segment. If ambiguous, ask the user.
  2. Call vss agent with the sensor id and ask for it to call video_understanding tool to answer the user's question.
  3. Return the vss agent's answer back to the user.

Query VSS agent (/generate)

# Set from deployment (compose / .env / host where vss-agent listens)
export VSS_AGENT_BASE_URL="http://localhost:8000"

curl -s -X POST "${VSS_AGENT_BASE_URL}/generate" \
  -H "Content-Type: application/json" \
  -d '{"input_message": "Call video_understanding tool to answer the following question about <sensor-id>: <user query>"}' | jq .

Response contract and extraction

/generate returns a JSON object with the assistant output in value, for example:

{"value":"<agent-think><agent-think-step ...>...</agent-think-step></agent-think>\n\n<final answer>\n\n"}

There is no separate clean-answer field. The consumable answer is the text in .value after removing any <agent-think>...</agent-think> block.

Required handling for this skill (and any downstream caller):

  1. Read .value from the JSON response.
  2. Strip <agent-think>...</agent-think> sections wherever they appear.
  3. Return only the remaining final-answer text to the user.

Example extraction:

curl -s -X POST "${VSS_AGENT_BASE_URL}/generate" \
  -H "Content-Type: application/json" \
  -d '{"input_message":"Call video_understanding tool to answer the following question about <sensor-id>: <user query>"}' \
| jq -r '.value' \
| python3 -c 'import re,sys; t=sys.stdin.read(); t=re.sub(r"<agent-think>.*?</agent-think>\s*", "", t, flags=re.S); print(t.strip())'

Cross-Reference

  • vss-manage-video-io-storage — VST storage/replay URLs so VIDEO_URL is valid for the VLM.
  • vss-generate-video-report — timestamped reports via Mode A (direct VLM) or Mode B (video-analytics incidents); this skill is VSS-agent /generate for ad-hoc video Q&A.

04 同仓库更多 Skills

Solizardking
Solizardking/skillhub-main

zkrouter

Self-hosted OpenAI-compatible LLM router — smart tier-based routing, ZK-stamped routing decisions, mode overrides, free OpenRouter at install via the Birth bot. Save 60-80% on inference costs by routing to the cheapest capable model across 12+ OpenRouter models.

★ 0
Solizardking
Solizardking/skillhub-main

zk

For custom ZK Solana programs and privacy-preserving applications to prevent double spending. Guide to integrate rent-free nullifier PDAs for double-spend prevention.

★ 0
Solizardking
Solizardking/skillhub-main

zk-omni-messaging

Zero-knowledge omnichain messaging between Robinhood Chain and Solana using CheshireZkOmniMessenger (msgType 4) with nullifier anti-replay and the zk-omni-relayer service.

★ 0
Solizardking
Solizardking/skillhub-main

zerion-wallet-data

Use when the user asks to check a wallet's portfolio, token balances, DeFi positions, PnL, transaction history, or NFTs. Provides interpreted on-chain data across 40+ EVM chains and Solana via x402 pay-per-request or API key — no raw RPC decoding needed.

★ 0
Solizardking
Solizardking/skillhub-main

youtube-clipper

YouTube 视频智能剪辑工具。下载视频和字幕,AI 分析生成精细章节(几分钟级别), 用户选择片段后自动剪辑、翻译字幕为中英双语、烧录字幕到视频,并生成总结文案。 使用场景:当用户需要剪辑 YouTube 视频、生成短视频片段、制作双语字幕版本时。 关键词:视频剪辑、YouTube、字幕翻译、双语字幕、视频下载、clip video

★ 0
Solizardking
Solizardking/skillhub-main

yield-optimization

Multi-chain yield optimization via yield.xyz (StakeKit). Discover 2,988+ yield opportunities across 75+ blockchains, build deposit/withdrawal transactions with shell scripts, and sign them with a MoonPay wallet. Use when the user wants to earn yield, find the best lending or staking rate, enter/exit a position, or check portfolio balances.

★ 0