[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M71PW66RWRVRMB7WP4NSTNMW":3,"skill-same:01M71PW66RWRVRMB7WP4NSTNMW":138},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":26,"publishedAt":47,"linkOnly":48,"markdownContent":49,"markdownBody":50,"frontMatter":51,"refName":59,"commitSha":60,"skillDirectory":61,"forksCount":14,"repositoryHtmlUrl":62,"repositoryHomepageUrl":63,"ownerHtmlUrl":64,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":65,"install":66,"resources":79},"01M71PW66RWRVRMB7WP4NSTNMW","tao-mine-aoi-images",null,"Runs the DEFT embed-then-mine workflow for VCN AOI iterations — embeds the gap-analysis target parquet, embeds a source pool, and mines nearest-neighbour source images for downstream augmentation. Use as the immediate next step after `tao-route-visual-changenet-samples` when expanding a real-image augmentation queue from the mining subset.","pattanation\u002FNorvik","pattanation","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F235419268?v=4","01M71M5THHYYB15ME8R2E4FCZG","Patta","01M71M5TJM1NDM992M8PDKYR2K",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Ftao-mine-aoi-images\u002FSKILL.md",[18,23],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",{"categoryNo":24,"categoryName":25,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发",[27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46],"tao mine aoi images","mine","tao","aoi","images","data","bank","read","at","next","gap-analysis","mining","route","analysis","Bash","image","workflow","when","sda","embedding","2026-11-10 13:02:02",false,"---\nname: tao-mine-aoi-images\ndescription: Runs the DEFT embed-then-mine workflow for VCN AOI iterations — embeds the gap-analysis target parquet, embeds a source pool, and mines nearest-neighbour source images for downstream augmentation. Use as the immediate next step after `tao-route-visual-changenet-samples` when expanding a real-image augmentation queue from the mining subset.\nlicense: Apache-2.0\ncompatibility: Requires docker + nvidia-container-toolkit and a CUDA GPU. Pulls the `tao_toolkit.data_services` image declared in `versions.yaml` at the skill bank root.\nmetadata:\n  author: NVIDIA Corporation\n  version: \"0.1.0\"\nallowed-tools: Read Bash\ntags:\n- data\n- mining\n- embedding\n- vcn\n- aoi\n- sda\n---\n\n# DEFT Mining and Embedding Skill\n\nYou are the operator of the DEFT embed-then-mine workflow for VCN AOI. Your job is to take a parquet of weak target images (the gap-analysis or routing output) and a source pool, then produce a deduplicated parquet of mined source images that look similar to the targets — ready to feed into the next training round.\n\nThe workflow is fixed and deterministic: **embed the targets, embed the source pool, then mine nearest neighbours.** Each step's output parquet is the next step's input. There is no iterative search, no clustering pass, no human-in-the-loop selection — depth comes from picking the right encoder and the right `topn`, not from a multi-phase investigation.\n\nThe whole skill is a thin wrapper around three direct `docker run` invocations against the `tao_toolkit.data_services` image declared in `versions.yaml` (resolved at runtime — see Setup). The container's entrypoint takes `\u003Ccategory> \u003Caction> -e \u003Cspec.yaml> [hydra overrides...]` — pass `embedding image_embeddings -e \u003Cembedding_spec.yaml> …` for embedding and `tmm nearest_neighbors -e \u003Cmining_spec.yaml> …` for mining. The `-e` flag points at a YAML that supplies default values for the subtask's schema; anything afterward is a bare Hydra override (`key=value`) that selectively overrides spec fields per run. (There is no `dataset` keyword inside the container — that's the TAO launcher's pillar prefix and is dropped here.) Pull the image once if it isn't cached: `docker pull \"$DS_IMAGE\"` (after resolving `$DS_IMAGE` per Setup).\n\nSchema keys can rename between data-services releases (the RCA skill saw `inference_csv` → `inference_results_dir`, `output_dir` → `results_dir`). When in doubt, introspect the actual schema once per image: `docker run --rm \"$DS_IMAGE\" embedding image_embeddings --cfg=job` and `... tmm nearest_neighbors --cfg=job`.\n\n---\n\n## Inputs\n\n1. **Target parquet** — the gap-analysis output, typically `mining_gaps.parquet` from `tao-route-visual-changenet-samples` (or `gaps.parquet` from `tao-analyze-gaps-visual-changenet` if routing was skipped). Required column: `filepath`. If `label` is also present, label-aware filtering during mining is available; otherwise the mining task silently no-ops the filter.\n2. **Source pool** — a parquet of candidate images to mine against, with a `filepath` column. If the user only has a CSV, convert it to a parquet **with the same columns** before Step 2. For label-aware filtering, the pool must also carry a `label` column.\n3. **Embedding spec file** — a YAML containing `model`, `model_path`, `batch_size`, and (only when `model_path` is a TAO `.pth`\u002F`.ckpt`) `model_config_path`. Reused across Steps 1 and 2; `input_parquet`\u002F`output_parquet` are supplied per run as Hydra overrides. The **same** spec MUST drive both embedding steps — embeddings from different encoders are not comparable, and mismatched encoders are the most common cause of \"the mined images look unrelated\" reports.\n4. **Mining spec file** — a YAML containing `topn`, `knn_metric`, `filter_by_label`, and (rarely changed) `source_embed_column_name`\u002F`target_embed_column_name`. `source_parquet`\u002F`target_parquet`\u002F`output_parquet` are Hydra overrides at run time. SigLIP and CLIP embeddings should use `knn_metric: cosine`. When `filter_by_label: true` but either embedding parquet lacks a `label` column, the container logs a warning and proceeds **without** filtering.\n\n---\n\n## Setup\n\nResolve the concrete `tao_toolkit.data_services` URI from `versions.yaml` once at the top of the run, then confirm Docker, the NVIDIA container toolkit, and a GPU are present before doing anything else. A GPU is required for both the encoder forward pass and the cuML\u002FcuDF k-NN search; both steps fail without CUDA.\n\n```bash\n# Resolve tao_toolkit.data_services → concrete nvcr.io\u002F... URI from versions.yaml\nDS_IMAGE=$(python3 -c \"import yaml,os; print(yaml.safe_load(open(os.environ['TAO_SKILL_BANK_PATH']+'\u002Fversions.yaml'))['images']['tao_toolkit']['data_services'])\")\necho \"DS_IMAGE=$DS_IMAGE\"\n\ndocker info > \u002Fdev\u002Fnull && echo \"OK: docker\"\nnvidia-smi > \u002Fdev\u002Fnull && echo \"OK: GPU\"\ndocker image inspect \"$DS_IMAGE\" > \u002Fdev\u002Fnull \\\n  || docker pull \"$DS_IMAGE\"\n```\n\nEvery host path the container reads or writes must be bind-mounted. The most predictable approach mounts the workspace root with **identical paths** inside and outside the container, then reuses one `$DOCKER` alias for the three invocations:\n\n```bash\nWORKSPACE=\u003Cabsolute path that contains all parquets, outputs, and the source-pool images>\nDOCKER=\"docker run --gpus all --rm --ipc=host -v $WORKSPACE:$WORKSPACE -w $WORKSPACE $DS_IMAGE\"\n```\n\nDo **not** pass `--user $(id -u):$(id -g)` — it triggers a `getpwuid()` `KeyError` during the `transformers` import before any work starts. The container runs as root; chown outputs back to the host UID afterward.\n\nAuthor the two spec files once per iteration, placing them under `$WORKSPACE` so the `-e` argument resolves on both sides of the mount; per-run values stay out of the spec and are passed as Hydra overrides. If the source pool is a CSV, convert it to parquet up front (preserving `filepath`, and `label` if present). The default `embedding_spec.yaml` uses `model: SigLIP`, `model_path: google\u002Fsiglip-base-patch16-224`, `batch_size: 64`; the default `mining_spec.yaml` uses `topn: 5`, `knn_metric: cosine`, `filter_by_label: \"false\"` (quoted — the schema reads it as a string).\n\nSee `references\u002Fsetup.md` for the full environment notes, `TAO_SKILL_BANK_PATH` handling, the path-mounting rationale, the `getpwuid` chown workaround, the CSV-to-parquet snippet, and the verbatim spec-file authoring blocks.\n\n---\n\n## Method\n\nThree commands, in order. Each command's output parquet is the next command's input. Run them as plain Bash; the `$DOCKER` alias from Setup handles the container, GPU, and mounts. Every invocation follows the same shape: `-e \u003Cspec>` for the baked-in defaults, then a handful of Hydra overrides for the run-specific paths.\n\n### Step 1 — Embed the target images\n\n```bash\n$DOCKER embedding image_embeddings \\\n    -e \u003Cembedding_spec.yaml> \\\n    input_parquet=\u003Ctarget_parquet> \\\n    output_parquet=\u003Ctarget_embeddings_parquet>\n```\n\nReads the gap-analysis \u002F routing output and writes a parquet with `filepath`, `embedding`, and any extra metadata columns (e.g. `label`, `siamese_score`, `weakness`) carried forward verbatim from the input. Print the output schema (`pd.read_parquet(...).columns`) to stdout so the script-check hook can confirm the embedding column exists.\n\nIf you need to override `model` \u002F `model_path` \u002F `batch_size` for one run without editing the spec, append them as Hydra overrides (e.g. `model_path=...`).\n\n### Step 2 — Embed the source pool\n\n```bash\n$DOCKER embedding image_embeddings \\\n    -e \u003Cembedding_spec.yaml> \\\n    input_parquet=\u003Csource_pool_parquet> \\\n    output_parquet=\u003Csource_embeddings_parquet>\n```\n\nSame command shape as Step 1, applied to the source pool. Use the **identical** `embedding_spec.yaml` as Step 1, and do not override `model` \u002F `model_path` \u002F `batch_size` differently here — mismatched encoder configs across the two steps produce non-comparable embeddings.\n\n### Step 3 — Mine nearest neighbours\n\n```bash\n$DOCKER tmm nearest_neighbors \\\n    -e \u003Cmining_spec.yaml> \\\n    source_parquet=\u003Csource_embeddings_parquet> \\\n    target_parquet=\u003Ctarget_embeddings_parquet> \\\n    output_parquet=\u003Cmined_parquet>\n```\n\nFor each target embedding, finds the `topn` closest source embeddings under the chosen metric, deduplicates across targets, and writes a single-column (`filepath`) parquet of unique mined source paths. The container also drops a `mining_summary.txt` next to the output parquet with: query count, neighbour count, duplicates removed, and (when label filtering is on) kept-vs-dropped pair counts. Tweak `topn`, `knn_metric`, or `filter_by_label` via inline Hydra override when sweeping (e.g. `topn=10`) — no need to rewrite the spec.\n\nWhen `filter_by_label=true` but one of the embedding parquets is missing the `label` column, the container logs a warning and proceeds without filtering. If the mined output looks larger than expected or contains cross-label pairs, scan the docker log for that warning before assuming the task did the right thing.\n\nSee `references\u002Freference-invocation.md` for the minimal paste-and-edit end-to-end recipe (resolves `$DS_IMAGE`, writes both specs, runs all three steps, chowns outputs, and prints row counts) to run as a single streamed Bash block.\n\n---\n\n## Outputs and report\n\nWrite everything into a timestamped folder under the experiment \u002F iteration directory. Get the real timestamp by running `date +%Y-%m-%d_%H%M%S` in Bash — do NOT hardcode or guess. If the user specifies a custom output path, use it directly but maintain the same internal layout. The packaging hook adds `mining_config\u002F` and `claude_session.jsonl` automatically when `Mining_Report.md` is written.\n\nThe mined parquet is the artifact downstream training consumes. The two embedding parquets are intermediate but worth retaining — reusable across multiple mining runs against the same source pool, and the only place to look when a \"looks unrelated\" report needs encoder-level debugging.\n\nSee `references\u002Foutputs-and-reporting.md` for the full output-directory layout and the verbatim `Mining_Report.md` template (Verdict, Inputs, Encoder Consistency, Mining Run, Per-Label Breakdown, Output Sanity, Recommended Actions; keep it 600–1200 words).\n\n---\n\n## Common pitfalls\n\nThe most frequent failure is **mismatched encoders between the two embedding steps** — the single most common cause of garbage mining output; both steps must consume the same `embedding_spec.yaml`. Other recurring traps: passing `--user` (the `getpwuid` `KeyError`), skipping an embedding step, a missing `label` column silently no-oping `filter_by_label=true`, spec files outside `$WORKSPACE`, unresolved `???` sentinels, TAO checkpoints without `model_config_path`, CSV source pools fed in directly, host\u002Fcontainer path mismatches, no GPU, an unpulled or `:latest` image tag, and `topn × N_targets ≫ source size` (expected — report the actual mined count).\n\nSee `references\u002Ftroubleshooting.md` for the full pitfall list with the exact errors, causes, and fixes.\n\n---\n\n## Execution Order\n\n1. Resolve `DS_IMAGE` from `versions.yaml` (`images.tao_toolkit.data_services`), then run `docker info`, `nvidia-smi`, and `docker image inspect \"$DS_IMAGE\"` (pulling if missing) once to confirm the environment. Abort with a clear message if any fail.\n2. Run `date +%Y-%m-%d_%H%M%S` to get the timestamp; create `\u003Coutput_dir>\u002Fmining_results\u002F\u003Ctimestamp>\u002F`.\n3. Write `embedding_spec.yaml` and `mining_spec.yaml` into the timestamped dir, filling in the encoder choice and mining knobs. Keep these under `$WORKSPACE` so the `-e` path resolves inside the container.\n4. If the source pool is a CSV, convert to parquet first (preserve `filepath` and `label`).\n5. Run Step 1 (embed targets) via `docker run … embedding image_embeddings -e embedding_spec.yaml input_parquet=… output_parquet=…`. Print the output parquet's row count and columns to stdout.\n6. Run Step 2 (embed source pool) with the **identical** `embedding_spec.yaml` as Step 1. Print output row count and columns.\n7. Run Step 3 (mine nearest neighbours) via `docker run … tmm nearest_neighbors -e mining_spec.yaml source_parquet=… target_parquet=… output_parquet=…`. Confirm `mining_summary.txt` was written next to `mined.parquet`.\n8. Compute the per-label breakdown (Section 5) by joining the target embeddings parquet with the mined output on filepath, if both carry `label`.\n9. Write `Mining_Report.md` last — writing it triggers the packaging hook, which copies session logs and skill config alongside.\n","# DEFT Mining and Embedding Skill\n\nYou are the operator of the DEFT embed-then-mine workflow for VCN AOI. Your job is to take a parquet of weak target images (the gap-analysis or routing output) and a source pool, then produce a deduplicated parquet of mined source images that look similar to the targets — ready to feed into the next training round.\n\nThe workflow is fixed and deterministic: **embed the targets, embed the source pool, then mine nearest neighbours.** Each step's output parquet is the next step's input. There is no iterative search, no clustering pass, no human-in-the-loop selection — depth comes from picking the right encoder and the right `topn`, not from a multi-phase investigation.\n\nThe whole skill is a thin wrapper around three direct `docker run` invocations against the `tao_toolkit.data_services` image declared in `versions.yaml` (resolved at runtime — see Setup). The container's entrypoint takes `\u003Ccategory> \u003Caction> -e \u003Cspec.yaml> [hydra overrides...]` — pass `embedding image_embeddings -e \u003Cembedding_spec.yaml> …` for embedding and `tmm nearest_neighbors -e \u003Cmining_spec.yaml> …` for mining. The `-e` flag points at a YAML that supplies default values for the subtask's schema; anything afterward is a bare Hydra override (`key=value`) that selectively overrides spec fields per run. (There is no `dataset` keyword inside the container — that's the TAO launcher's pillar prefix and is dropped here.) Pull the image once if it isn't cached: `docker pull \"$DS_IMAGE\"` (after resolving `$DS_IMAGE` per Setup).\n\nSchema keys can rename between data-services releases (the RCA skill saw `inference_csv` → `inference_results_dir`, `output_dir` → `results_dir`). When in doubt, introspect the actual schema once per image: `docker run --rm \"$DS_IMAGE\" embedding image_embeddings --cfg=job` and `... tmm nearest_neighbors --cfg=job`.\n\n---\n\n## Inputs\n\n1. **Target parquet** — the gap-analysis output, typically `mining_gaps.parquet` from `tao-route-visual-changenet-samples` (or `gaps.parquet` from `tao-analyze-gaps-visual-changenet` if routing was skipped). Required column: `filepath`. If `label` is also present, label-aware filtering during mining is available; otherwise the mining task silently no-ops the filter.\n2. **Source pool** — a parquet of candidate images to mine against, with a `filepath` column. If the user only has a CSV, convert it to a parquet **with the same columns** before Step 2. For label-aware filtering, the pool must also carry a `label` column.\n3. **Embedding spec file** — a YAML containing `model`, `model_path`, `batch_size`, and (only when `model_path` is a TAO `.pth`\u002F`.ckpt`) `model_config_path`. Reused across Steps 1 and 2; `input_parquet`\u002F`output_parquet` are supplied per run as Hydra overrides. The **same** spec MUST drive both embedding steps — embeddings from different encoders are not comparable, and mismatched encoders are the most common cause of \"the mined images look unrelated\" reports.\n4. **Mining spec file** — a YAML containing `topn`, `knn_metric`, `filter_by_label`, and (rarely changed) `source_embed_column_name`\u002F`target_embed_column_name`. `source_parquet`\u002F`target_parquet`\u002F`output_parquet` are Hydra overrides at run time. SigLIP and CLIP embeddings should use `knn_metric: cosine`. When `filter_by_label: true` but either embedding parquet lacks a `label` column, the container logs a warning and proceeds **without** filtering.\n\n---\n\n## Setup\n\nResolve the concrete `tao_toolkit.data_services` URI from `versions.yaml` once at the top of the run, then confirm Docker, the NVIDIA container toolkit, and a GPU are present before doing anything else. A GPU is required for both the encoder forward pass and the cuML\u002FcuDF k-NN search; both steps fail without CUDA.\n\n```bash\n# Resolve tao_toolkit.data_services → concrete nvcr.io\u002F... URI from versions.yaml\nDS_IMAGE=$(python3 -c \"import yaml,os; print(yaml.safe_load(open(os.environ['TAO_SKILL_BANK_PATH']+'\u002Fversions.yaml'))['images']['tao_toolkit']['data_services'])\")\necho \"DS_IMAGE=$DS_IMAGE\"\n\ndocker info > \u002Fdev\u002Fnull && echo \"OK: docker\"\nnvidia-smi > \u002Fdev\u002Fnull && echo \"OK: GPU\"\ndocker image inspect \"$DS_IMAGE\" > \u002Fdev\u002Fnull \\\n  || docker pull \"$DS_IMAGE\"\n```\n\nEvery host path the container reads or writes must be bind-mounted. The most predictable approach mounts the workspace root with **identical paths** inside and outside the container, then reuses one `$DOCKER` alias for the three invocations:\n\n```bash\nWORKSPACE=\u003Cabsolute path that contains all parquets, outputs, and the source-pool images>\nDOCKER=\"docker run --gpus all --rm --ipc=host -v $WORKSPACE:$WORKSPACE -w $WORKSPACE $DS_IMAGE\"\n```\n\nDo **not** pass `--user $(id -u):$(id -g)` — it triggers a `getpwuid()` `KeyError` during the `transformers` import before any work starts. The container runs as root; chown outputs back to the host UID afterward.\n\nAuthor the two spec files once per iteration, placing them under `$WORKSPACE` so the `-e` argument resolves on both sides of the mount; per-run values stay out of the spec and are passed as Hydra overrides. If the source pool is a CSV, convert it to parquet up front (preserving `filepath`, and `label` if present). The default `embedding_spec.yaml` uses `model: SigLIP`, `model_path: google\u002Fsiglip-base-patch16-224`, `batch_size: 64`; the default `mining_spec.yaml` uses `topn: 5`, `knn_metric: cosine`, `filter_by_label: \"false\"` (quoted — the schema reads it as a string).\n\nSee `references\u002Fsetup.md` for the full environment notes, `TAO_SKILL_BANK_PATH` handling, the path-mounting rationale, the `getpwuid` chown workaround, the CSV-to-parquet snippet, and the verbatim spec-file authoring blocks.\n\n---\n\n## Method\n\nThree commands, in order. Each command's output parquet is the next command's input. Run them as plain Bash; the `$DOCKER` alias from Setup handles the container, GPU, and mounts. Every invocation follows the same shape: `-e \u003Cspec>` for the baked-in defaults, then a handful of Hydra overrides for the run-specific paths.\n\n### Step 1 — Embed the target images\n\n```bash\n$DOCKER embedding image_embeddings \\\n    -e \u003Cembedding_spec.yaml> \\\n    input_parquet=\u003Ctarget_parquet> \\\n    output_parquet=\u003Ctarget_embeddings_parquet>\n```\n\nReads the gap-analysis \u002F routing output and writes a parquet with `filepath`, `embedding`, and any extra metadata columns (e.g. `label`, `siamese_score`, `weakness`) carried forward verbatim from the input. Print the output schema (`pd.read_parquet(...).columns`) to stdout so the script-check hook can confirm the embedding column exists.\n\nIf you need to override `model` \u002F `model_path` \u002F `batch_size` for one run without editing the spec, append them as Hydra overrides (e.g. `model_path=...`).\n\n### Step 2 — Embed the source pool\n\n```bash\n$DOCKER embedding image_embeddings \\\n    -e \u003Cembedding_spec.yaml> \\\n    input_parquet=\u003Csource_pool_parquet> \\\n    output_parquet=\u003Csource_embeddings_parquet>\n```\n\nSame command shape as Step 1, applied to the source pool. Use the **identical** `embedding_spec.yaml` as Step 1, and do not override `model` \u002F `model_path` \u002F `batch_size` differently here — mismatched encoder configs across the two steps produce non-comparable embeddings.\n\n### Step 3 — Mine nearest neighbours\n\n```bash\n$DOCKER tmm nearest_neighbors \\\n    -e \u003Cmining_spec.yaml> \\\n    source_parquet=\u003Csource_embeddings_parquet> \\\n    target_parquet=\u003Ctarget_embeddings_parquet> \\\n    output_parquet=\u003Cmined_parquet>\n```\n\nFor each target embedding, finds the `topn` closest source embeddings under the chosen metric, deduplicates across targets, and writes a single-column (`filepath`) parquet of unique mined source paths. The container also drops a `mining_summary.txt` next to the output parquet with: query count, neighbour count, duplicates removed, and (when label filtering is on) kept-vs-dropped pair counts. Tweak `topn`, `knn_metric`, or `filter_by_label` via inline Hydra override when sweeping (e.g. `topn=10`) — no need to rewrite the spec.\n\nWhen `filter_by_label=true` but one of the embedding parquets is missing the `label` column, the container logs a warning and proceeds without filtering. If the mined output looks larger than expected or contains cross-label pairs, scan the docker log for that warning before assuming the task did the right thing.\n\nSee `references\u002Freference-invocation.md` for the minimal paste-and-edit end-to-end recipe (resolves `$DS_IMAGE`, writes both specs, runs all three steps, chowns outputs, and prints row counts) to run as a single streamed Bash block.\n\n---\n\n## Outputs and report\n\nWrite everything into a timestamped folder under the experiment \u002F iteration directory. Get the real timestamp by running `date +%Y-%m-%d_%H%M%S` in Bash — do NOT hardcode or guess. If the user specifies a custom output path, use it directly but maintain the same internal layout. The packaging hook adds `mining_config\u002F` and `claude_session.jsonl` automatically when `Mining_Report.md` is written.\n\nThe mined parquet is the artifact downstream training consumes. The two embedding parquets are intermediate but worth retaining — reusable across multiple mining runs against the same source pool, and the only place to look when a \"looks unrelated\" report needs encoder-level debugging.\n\nSee `references\u002Foutputs-and-reporting.md` for the full output-directory layout and the verbatim `Mining_Report.md` template (Verdict, Inputs, Encoder Consistency, Mining Run, Per-Label Breakdown, Output Sanity, Recommended Actions; keep it 600–1200 words).\n\n---\n\n## Common pitfalls\n\nThe most frequent failure is **mismatched encoders between the two embedding steps** — the single most common cause of garbage mining output; both steps must consume the same `embedding_spec.yaml`. Other recurring traps: passing `--user` (the `getpwuid` `KeyError`), skipping an embedding step, a missing `label` column silently no-oping `filter_by_label=true`, spec files outside `$WORKSPACE`, unresolved `???` sentinels, TAO checkpoints without `model_config_path`, CSV source pools fed in directly, host\u002Fcontainer path mismatches, no GPU, an unpulled or `:latest` image tag, and `topn × N_targets ≫ source size` (expected — report the actual mined count).\n\nSee `references\u002Ftroubleshooting.md` for the full pitfall list with the exact errors, causes, and fixes.\n\n---\n\n## Execution Order\n\n1. Resolve `DS_IMAGE` from `versions.yaml` (`images.tao_toolkit.data_services`), then run `docker info`, `nvidia-smi`, and `docker image inspect \"$DS_IMAGE\"` (pulling if missing) once to confirm the environment. Abort with a clear message if any fail.\n2. Run `date +%Y-%m-%d_%H%M%S` to get the timestamp; create `\u003Coutput_dir>\u002Fmining_results\u002F\u003Ctimestamp>\u002F`.\n3. Write `embedding_spec.yaml` and `mining_spec.yaml` into the timestamped dir, filling in the encoder choice and mining knobs. Keep these under `$WORKSPACE` so the `-e` path resolves inside the container.\n4. If the source pool is a CSV, convert to parquet first (preserve `filepath` and `label`).\n5. Run Step 1 (embed targets) via `docker run … embedding image_embeddings -e embedding_spec.yaml input_parquet=… output_parquet=…`. Print the output parquet's row count and columns to stdout.\n6. Run Step 2 (embed source pool) with the **identical** `embedding_spec.yaml` as Step 1. Print output row count and columns.\n7. Run Step 3 (mine nearest neighbours) via `docker run … tmm nearest_neighbors -e mining_spec.yaml source_parquet=… target_parquet=… output_parquet=…`. Confirm `mining_summary.txt` was written next to `mined.parquet`.\n8. Compute the per-label breakdown (Section 5) by joining the target embeddings parquet with the mined output on filepath, if both carry `label`.\n9. Write `Mining_Report.md` last — writing it triggers the packaging hook, which copies session logs and skill config alongside.\n",{"name":5,"tags":52,"license":15,"metadata":54,"description":7,"allowed-tools":57,"compatibility":58},[32,38,46,53,30,45],"vcn",{"author":55,"version":56},"NVIDIA Corporation","0.1.0","Read Bash","Requires docker + nvidia-container-toolkit and a CUDA GPU. Pulls the `tao_toolkit.data_services` image declared in `versions.yaml` at the skill bank root.","claude\u002Finstall-gsd-build-foQdV","d86cd127f527124b5fd0fd0bc679a4d79b7d01d7",".agents\u002Fskills\u002Ftao-mine-aoi-images","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik","https:\u002F\u002Fnorvik-orpin.vercel.app","https:\u002F\u002Fgithub.com\u002Fpattanation","2026-07-17 11:17:04",{"source":8,"skillName":5,"commands":67,"prompt":77,"zipAvailable":78},[68,71,74],{"tool":69,"command":70},"npx","npx skills add pattanation\u002FNorvik --skill tao-mine-aoi-images",{"tool":72,"command":73},"bunx","bunx skills add pattanation\u002FNorvik --skill tao-mine-aoi-images",{"tool":75,"command":76},"pnpm","pnpm dlx skills add pattanation\u002FNorvik --skill tao-mine-aoi-images","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Ftao-mine-aoi-images\u002FSKILL.md",true,[80,86,90,96,100,104,108,112,116,120,124,128,132],{"path":81,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":84,"sourceHtmlUrl":85},"BENCHMARK.md","REFERENCE","参考文档",4053,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Ftao-mine-aoi-images\u002FBENCHMARK.md",{"path":87,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":88,"sourceHtmlUrl":89},"evals\u002Fevals.json",941,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Ftao-mine-aoi-images\u002Fevals\u002Fevals.json",{"path":91,"resourceType":92,"resourceTypeDesc":93,"sizeBytes":94,"sourceHtmlUrl":95},"hooks\u002F_parse-stdin.sh","SCRIPT","脚本",1471,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Ftao-mine-aoi-images\u002Fhooks\u002F_parse-stdin.sh",{"path":97,"resourceType":92,"resourceTypeDesc":93,"sizeBytes":98,"sourceHtmlUrl":99},"hooks\u002Fmining-artifacts-check.sh",3524,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Ftao-mine-aoi-images\u002Fhooks\u002Fmining-artifacts-check.sh",{"path":101,"resourceType":92,"resourceTypeDesc":93,"sizeBytes":102,"sourceHtmlUrl":103},"hooks\u002Fmining-package.sh",2557,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Ftao-mine-aoi-images\u002Fhooks\u002Fmining-package.sh",{"path":105,"resourceType":92,"resourceTypeDesc":93,"sizeBytes":106,"sourceHtmlUrl":107},"hooks\u002Fmining-script-check.sh",5283,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Ftao-mine-aoi-images\u002Fhooks\u002Fmining-script-check.sh",{"path":109,"resourceType":92,"resourceTypeDesc":93,"sizeBytes":110,"sourceHtmlUrl":111},"hooks\u002Fmining-section-check.sh",2886,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Ftao-mine-aoi-images\u002Fhooks\u002Fmining-section-check.sh",{"path":113,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":114,"sourceHtmlUrl":115},"references\u002Foutputs-and-reporting.md",4419,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Ftao-mine-aoi-images\u002Freferences\u002Foutputs-and-reporting.md",{"path":117,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":118,"sourceHtmlUrl":119},"references\u002Freference-invocation.md",2766,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Ftao-mine-aoi-images\u002Freferences\u002Freference-invocation.md",{"path":121,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":122,"sourceHtmlUrl":123},"references\u002Fsetup.md",3600,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Ftao-mine-aoi-images\u002Freferences\u002Fsetup.md",{"path":125,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":126,"sourceHtmlUrl":127},"references\u002Ftroubleshooting.md",3224,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Ftao-mine-aoi-images\u002Freferences\u002Ftroubleshooting.md",{"path":129,"resourceType":82,"resourceTypeDesc":83,"sizeBytes":130,"sourceHtmlUrl":131},"skill-card.md",3850,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Ftao-mine-aoi-images\u002Fskill-card.md",{"path":133,"resourceType":134,"resourceTypeDesc":135,"sizeBytes":136,"sourceHtmlUrl":137},"skill.oms.sig","OTHER","其他",6757,"https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Ftao-mine-aoi-images\u002Fskill.oms.sig",[139,170,196,220,242,256],{"skillNo":140,"name":141,"nameZh":6,"description":142,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":143,"sourceHtmlUrl":144,"categories":145,"keywords":152,"publishedAt":47},"01M71QDV4DM8T19HEHWATEF1K4","webapp-testing","Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fwebapp-testing\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fwebapp-testing\u002FSKILL.md",[146,149],{"categoryNo":147,"categoryName":148,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000008","测试与质量",{"categoryNo":150,"categoryName":151,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000009","前端开发",[153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169],"webapp","testing","webapp testing","for","applications","local","frontend","debugging","toolkit","logs","browser","with","playwright","web","behavior","ui","screenshots",{"skillNo":171,"name":172,"nameZh":6,"description":173,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":174,"sourceHtmlUrl":175,"categories":176,"keywords":178,"publishedAt":47},"01M71QDH2GGMXGA9105C9W1N7B","web-artifacts-builder","Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn\u002Fui). Use for complex artifacts requiring state management, routing, or shadcn\u002Fui components - not for simple single-file HTML\u002FJSX artifacts.","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fweb-artifacts-builder\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fweb-artifacts-builder\u002FSKILL.md",[177],{"categoryNo":150,"categoryName":151,"taxonomyType":21,"taxonomyTypeDesc":22},[179,180,181,166,182,183,184,156,185,186,159,187,188,189,190,191,192,193,194,195],"builder","artifacts builder","artifacts","web artifacts builder","single-file","css","Tailwind","ui components","single","use","file","components","react","multi","state","routing","state management",{"skillNo":197,"name":198,"nameZh":6,"description":199,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":200,"licenseFileUrl":201,"sourceHtmlUrl":202,"categories":203,"keywords":205,"publishedAt":47},"01M71QD9T2XC3EWMQNN9ACNWQ1","vueuse-functions","Apply VueUse composables where appropriate to build concise, maintainable Vue.js \u002F Nuxt features.","MIT","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvueuse-functions\u002FLICENSE.md","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvueuse-functions\u002FSKILL.md",[204],{"categoryNo":150,"categoryName":151,"taxonomyType":21,"taxonomyTypeDesc":22},[206,207,208,209,210,211,212,213,214,215,216,217,218,219],"functions","vueuse","vueuse functions","project","nuxt","apply","github","concise","build","https","features","vue","js","composables",{"skillNo":221,"name":222,"nameZh":6,"description":223,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":200,"licenseFileUrl":224,"sourceHtmlUrl":225,"categories":226,"keywords":229,"publishedAt":47},"01M71QD51HWSGYMCMB7RW7PGK8","vue-testing-best-practices","Use for Vue.js testing. Covers Vitest, Vue Test Utils, component testing, mocking, testing patterns, and Playwright for E2E testing.","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvue-testing-best-practices\u002FLICENSE.md","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvue-testing-best-practices\u002FSKILL.md",[227,228],{"categoryNo":147,"categoryName":148,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":150,"categoryName":151,"taxonomyType":21,"taxonomyTypeDesc":22},[230,154,231,217,232,165,233,234,235,212,236,156,237,238,239,188,240,218,241],"vue testing best practices","testing best practices","best practices","test","testing patterns","component","vitest","mocking","e2e","utils","patterns","e2e testing",{"skillNo":243,"name":244,"nameZh":6,"description":245,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":200,"licenseFileUrl":246,"sourceHtmlUrl":247,"categories":248,"keywords":250,"publishedAt":47},"01M71QD358H2DJ9THJRBP77T3T","vue-router-best-practices","Vue Router 4 patterns, navigation guards, route params, and route-component lifecycle interactions.","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvue-router-best-practices\u002FLICENSE.md","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvue-router-best-practices\u002FSKILL.md",[249],{"categoryNo":150,"categoryName":151,"taxonomyType":21,"taxonomyTypeDesc":22},[251,217,252,232,39,235,253,212,240,254,255],"router","vue router best practices","guards","navigation","lifecycle",{"skillNo":257,"name":258,"nameZh":6,"description":259,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":200,"licenseFileUrl":260,"sourceHtmlUrl":261,"categories":262,"keywords":264,"publishedAt":47},"01M71QCYPNWDE2MQZ1TF3ED5S4","vue-best-practices","MUST be used for Vue.js tasks. Strongly recommends Composition API with `\u003Cscript setup>` and TypeScript as the standard approach. Covers Vue 3, SSR, Volar, vue-tsc. Load for any Vue, .vue files, Vue Router, Pinia, or Vite with Vue work. ALWAYS use Composition API unless the project explicitly requires Options API.","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvue-best-practices\u002FLICENSE.md","https:\u002F\u002Fgithub.com\u002Fpattanation\u002FNorvik\u002Fblob\u002Fclaude\u002Finstall-gsd-build-foQdV\u002F.agents\u002Fskills\u002Fvue-best-practices\u002FSKILL.md",[263],{"categoryNo":150,"categoryName":151,"taxonomyType":21,"taxonomyTypeDesc":22},[265,217,232,266,209,267,268,269,251,270,271,272,273,274,275,156,276,188,277,278],"vue best practices","Composition","files","tasks","api","pinia","work","options","ssr","typescript","as","vite","load","standard"]