output-and-integrity
Safely plan and validate ncbi-genome-download output layout, checksums, cache behavior, retries, concurrency, metadata tables, and human-readable links without causing unintended genome downloads.
- License 许可
- Apache-2.0
- 最近更新
- 2026-10-08
- Stars
- ★ 11
- Forks
- 3
01 前言 Front Matter
- name
- output-and-integrity
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Safely plan and validate ncbi-genome-download output layout, checksums, cache behavior, retries, concurrency, metadata tables, and human-readable links without causing unintended genome downloads.
- disable-model-invocation
- true
02 资源文件 4
- 参考文档references/checksums-resume.md6.6 KB
- 参考文档references/metadata-and-concurrency.md6.1 KB
- 参考文档references/output-layout.md5.9 KB
- 参考文档references/troubleshooting.md8.2 KB
03 SKILL.md 正文
Output and integrity
Use this skill after the candidate set and filters are already known. It owns where files go, how an existing file is judged reusable, how MD5SUMS and metadata are handled, and how to recover from an interrupted or partial run. For selecting assemblies, formats, sections, and taxonomic filters, route to download-and-filter. For ETE3-based taxonomy expansion, route to taxonomy-helper.
The facts here target the installed ncbi-genome-download 0.3.4 package
(the package uses requests, tqdm, and appdirs). Prefer a dry run or a
mocked local response before any real network operation.
Safety boundary: know which phase has side effects
--dry-run/-nstill obtains and parses each assembly summary needed to select candidates. It prints accession, organism, and strain, returns0, and does not create output directories, fetchmd5checksums.txt, or fetch genome files. It can therefore still make summary-network requests unless a usable summary cache is explicitly enabled through the Python API.- A normal run fetches summaries, then checksum manifests, then selected files. It creates directories and may replace files and symlinks. Use a temporary output directory and mocked URLs for verification.
--no-cache/-Ndisables the assembly-summary cache. In 0.3.4 the CLI default is alsouse_cache=False(the Python API can passuse_cache=True); do not assume the README's cache wording overrides the installed defaults. This cache is separate from per-assemblyMD5SUMS.--md5-cache-days Ncontrols how long a nested-outputMD5SUMSfile is trusted. It does not make downloaded genome files expire, and it does not cache checksums in flat-output mode.
Before a real run, inspect the resolved config and do a dry run, for example:
ncbi-genome-download --dry-run --assembly-accessions GCF_000203835.1 \
--formats genbank --output-folder ./ngd-scratch bacteria
The accession above is only a selection example; do not remove --dry-run
unless the source and destination are intentionally approved. A small mocked
summary/checksum/file fixture should be used to test a normal run. Never use a
live NCBI URL in a validation example.
Operating procedure
- Route selection and filter questions to the sibling skill, then record the exact formats and destination. Decide nested versus flat before starting.
- Run a dry run and confirm that the candidate count is nonzero. An empty
candidate selection is a controlled result with return code
1, not a reason to broaden filters automatically. - For a fixture run, provide a local/mock HTTP response for the summary,
md5checksums.txt, and each selected file. Keep the output under a fresh temporary directory and use one format first. - Check the expected file path, the MD5 comparison, and (if requested) the metadata TSV. Only then consider parallelism or a larger approved run.
- On a rerun, let checksum comparison decide whether a file is reusable. Do not call a partial file “resumed”: this implementation has no HTTP range resume. A missing or mismatching file is downloaded again.
- If a connection exception escapes selection or a worker, the command uses
return code
75(temporary failure); retry the command with bounded--retries N. A checksum mismatch causes the worker to return false, but 0.3.4 does not aggregate worker booleans into the final status; inspect the log and files rather than treating code0as proof every file passed.
Lower-level API map
NgdConfig/NgdConfig.from_kwargs(...)holdsoutput,flat_output,human_readable,parallel,progress_bar,metadata_table,dry_run,use_cache, andmd5_cache_days.download(**kwargs)returns the same integer status as the CLI path.create_dir(entry, section, domain, output, flat_output)makes and returns the nested assembly directory or the output root in flat mode.create_readable_dir(...)makes the human-readable directory. It does not copy data; later jobs make links.DownloadJob(full_url, local_file, expected_checksum, symlink_path)is the unit consumed byworker. A normal job has a URL and expected MD5. A symlink-only job hasfull_url=Noneand does not belong in metadata rows.parse_checksums(text)returns dictionaries withchecksumandfile, ignores blank/malformed lines, and strips a leading./from filenames.md5sum(path)hashes the file in 4096-byte chunks.has_file_changed(...)returns true for a missing file or an MD5 mismatch.save_and_check(...)writes response chunks and then compares MD5; it is not atomic.metadata.get(columns=None)returns a process-globalMetaDataobject;metadata.clear()resets it.MetaData.add(entry, local_file)queues a row andMetaData.write(handle)emits TSV. Clear it between multiple API runs in one interpreter.
Read the focused references in this order: output layout, checksums and reruns, metadata and concurrency, and troubleshooting. They cross-link back here and to the two routing siblings.
Return-code and verification contract
| Status | Meaning in 0.3.4 | Safe response |
|---|---|---|
0 |
Candidate processing completed without a caught connection exception; dry runs also use it. | Inspect expected files, MD5, links, and TSV; do not infer that every worker returned true. |
1 |
No candidates matched, or a parallel run was interrupted by KeyboardInterrupt. |
Fix filters or preserve the partial output and rerun; do not delete valid files blindly. |
75 |
A requests connection/chunked-transfer failure reached config_download; the CLI may retry. |
Retry with a bounded count after checking connectivity and disk state. |
A ValueError raised while parsing CLI configuration is printed by the
entrypoint and may be returned as -2; this is an invalid invocation rather
than a download result. The normal operating checks required here remain the
three statuses above.
Bundled-script inventory
No bundled script is included. A checksum smoke script would duplicate a few
four-line fixture assertions and could encourage treating a successful helper
as proof of a network run. The self-contained fixture procedure in
checksums and reruns exercises parsing,
filename selection, MD5, save_and_check, output paths, and TSV without
network access; it is the safer reusable check.
04 同仓库更多 Skills
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
python-api
Programmatic Object-Detection-Metrics API guidance for BoundingBox, BoundingBoxes, Evaluator, enums, coordinate conversion, AP/mAP semantics, and source-style import diagnostics.
file-evaluation
Folder-based PASCAL VOC AP/mAP evaluation from ground-truth and detection text files.
object-detection-metrics
Guides legacy Object-Detection-Metrics PASCAL VOC AP/mAP evaluation from detection text folders and direct Python API classes.
nx-workspace-patterns
Configure and optimize Nx monorepo workspaces. Use when setting up Nx, configuring project boundaries, optimizing build caching, or implementing affected commands.