AGENT SKILL

output-and-integrity

Safely plan and validate ncbi-genome-download output layout, checksums, cache behavior, retries, concurrency, metadata tables, and human-readable links without causing unintended genome downloads.

License 许可
Apache-2.0
最近更新
2026-09-03
Stars
★ 351
Forks
27

01 前言 Front Matter

name
output-and-integrity
license
Apache 2.0
metadata
{ "disco-role": "operating" }
description
Safely plan and validate ncbi-genome-download output layout, checksums, cache behavior, retries, concurrency, metadata tables, and human-readable links without causing unintended genome downloads.
disable-model-invocation
true

02 资源文件 4

03 SKILL.md 正文

Output and integrity

Use this skill after the candidate set and filters are already known. It owns where files go, how an existing file is judged reusable, how MD5SUMS and metadata are handled, and how to recover from an interrupted or partial run. For selecting assemblies, formats, sections, and taxonomic filters, route to download-and-filter. For ETE3-based taxonomy expansion, route to taxonomy-helper.

The facts here target the installed ncbi-genome-download 0.3.4 package (the package uses requests, tqdm, and appdirs). Prefer a dry run or a mocked local response before any real network operation.

Safety boundary: know which phase has side effects

  • --dry-run/-n still obtains and parses each assembly summary needed to select candidates. It prints accession, organism, and strain, returns 0, and does not create output directories, fetch md5checksums.txt, or fetch genome files. It can therefore still make summary-network requests unless a usable summary cache is explicitly enabled through the Python API.
  • A normal run fetches summaries, then checksum manifests, then selected files. It creates directories and may replace files and symlinks. Use a temporary output directory and mocked URLs for verification.
  • --no-cache/-N disables the assembly-summary cache. In 0.3.4 the CLI default is also use_cache=False (the Python API can pass use_cache=True); do not assume the README's cache wording overrides the installed defaults. This cache is separate from per-assembly MD5SUMS.
  • --md5-cache-days N controls how long a nested-output MD5SUMS file is trusted. It does not make downloaded genome files expire, and it does not cache checksums in flat-output mode.

Before a real run, inspect the resolved config and do a dry run, for example:

ncbi-genome-download --dry-run --assembly-accessions GCF_000203835.1 \
  --formats genbank --output-folder ./ngd-scratch bacteria

The accession above is only a selection example; do not remove --dry-run unless the source and destination are intentionally approved. A small mocked summary/checksum/file fixture should be used to test a normal run. Never use a live NCBI URL in a validation example.

Operating procedure

  1. Route selection and filter questions to the sibling skill, then record the exact formats and destination. Decide nested versus flat before starting.
  2. Run a dry run and confirm that the candidate count is nonzero. An empty candidate selection is a controlled result with return code 1, not a reason to broaden filters automatically.
  3. For a fixture run, provide a local/mock HTTP response for the summary, md5checksums.txt, and each selected file. Keep the output under a fresh temporary directory and use one format first.
  4. Check the expected file path, the MD5 comparison, and (if requested) the metadata TSV. Only then consider parallelism or a larger approved run.
  5. On a rerun, let checksum comparison decide whether a file is reusable. Do not call a partial file “resumed”: this implementation has no HTTP range resume. A missing or mismatching file is downloaded again.
  6. If a connection exception escapes selection or a worker, the command uses return code 75 (temporary failure); retry the command with bounded --retries N. A checksum mismatch causes the worker to return false, but 0.3.4 does not aggregate worker booleans into the final status; inspect the log and files rather than treating code 0 as proof every file passed.

Lower-level API map

  • NgdConfig/NgdConfig.from_kwargs(...) holds output, flat_output, human_readable, parallel, progress_bar, metadata_table, dry_run, use_cache, and md5_cache_days. download(**kwargs) returns the same integer status as the CLI path.
  • create_dir(entry, section, domain, output, flat_output) makes and returns the nested assembly directory or the output root in flat mode.
  • create_readable_dir(...) makes the human-readable directory. It does not copy data; later jobs make links.
  • DownloadJob(full_url, local_file, expected_checksum, symlink_path) is the unit consumed by worker. A normal job has a URL and expected MD5. A symlink-only job has full_url=None and does not belong in metadata rows.
  • parse_checksums(text) returns dictionaries with checksum and file, ignores blank/malformed lines, and strips a leading ./ from filenames.
  • md5sum(path) hashes the file in 4096-byte chunks. has_file_changed(...) returns true for a missing file or an MD5 mismatch. save_and_check(...) writes response chunks and then compares MD5; it is not atomic.
  • metadata.get(columns=None) returns a process-global MetaData object; metadata.clear() resets it. MetaData.add(entry, local_file) queues a row and MetaData.write(handle) emits TSV. Clear it between multiple API runs in one interpreter.

Read the focused references in this order: output layout, checksums and reruns, metadata and concurrency, and troubleshooting. They cross-link back here and to the two routing siblings.

Return-code and verification contract

Status Meaning in 0.3.4 Safe response
0 Candidate processing completed without a caught connection exception; dry runs also use it. Inspect expected files, MD5, links, and TSV; do not infer that every worker returned true.
1 No candidates matched, or a parallel run was interrupted by KeyboardInterrupt. Fix filters or preserve the partial output and rerun; do not delete valid files blindly.
75 A requests connection/chunked-transfer failure reached config_download; the CLI may retry. Retry with a bounded count after checking connectivity and disk state.

A ValueError raised while parsing CLI configuration is printed by the entrypoint and may be returned as -2; this is an invalid invocation rather than a download result. The normal operating checks required here remain the three statuses above.

Bundled-script inventory

No bundled script is included. A checksum smoke script would duplicate a few four-line fixture assertions and could encourage treating a successful helper as proof of a network run. The self-contained fixture procedure in checksums and reruns exercises parsing, filename selection, MD5, save_and_check, output paths, and TSV without network access; it is the safer reusable check.

04 同仓库更多 Skills

VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-triton-opt

Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-solve

END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-skill

PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-pattern-fusion

Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-orchestrate

Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-feedback

Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).

★ 351