discovery-downloads
Find, filter, retrieve, and politely crawl web pages with Trafilatura discovery/download APIs and CLI flags.
- License 许可
- Apache-2.0
- 最近更新
- 2026-10-08
- Stars
- ★ 11
- Forks
- 3
01 前言 Front Matter
- name
- discovery-downloads
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Find, filter, retrieve, and politely crawl web pages with Trafilatura discovery/download APIs and CLI flags.
- disable-model-invocation
- true
02 资源文件 3
- 参考文档references/api-reference.md13.5 KB
- 参考文档references/troubleshooting.md13.3 KB
- 参考文档references/workflows.md15.6 KB
03 SKILL.md 正文
Trafilatura Discovery and Downloads
Use this sub-skill when the task is to find candidate URLs, fetch web pages, inspect HTTP responses, or run Trafilatura's supported feed/sitemap/focused-crawl navigation before extraction.
Best-fit tasks
- Fetch one page with
fetch_url()or inspect status/headers/final URL withfetch_response(). - Build a polite domain-aware download queue with
UrlStore,load_download_buffer(), and buffered download helpers. - Discover candidate article URLs through feeds, sitemaps, and the focused crawler.
- Use CLI navigation flags such as
--feed,--sitemap,--crawl,--explore,--probe,--archived,--url-filter, and--list. - Diagnose blocked downloads, SSL/proxy/timeouts, file-size limits, robots rules, and optional download extras.
Read these references
- API reference for signatures, return objects, response fields, URL store behavior, and discovery helper details.
- Workflows for ready-to-adapt Python and CLI recipes covering single downloads, batch queues, feeds, sitemaps, focused crawling, and Internet Archive fallback.
- Troubleshooting for network failures, polite throttling, proxies, SSL, optional extras, and long-tail limits.
Bundled validation
Run the offline smoke check before relying on this sub-skill in a new environment:
python skills/disco/trafilatura/sub-skills/discovery-downloads/scripts/discovery_smoke.py
The script imports the discovery/download APIs, asserts key signatures, exercises Response, offline feed/sitemap parsing, URL filtering/queue behavior, and a mocked focused-crawl step. It does not contact the network.
Route elsewhere
- Use the extraction/output sub-skills for HTML-to-text extraction, metadata/comments extraction, output formats, or stdin/file conversion after retrieval.
- Use the corpus-quality/deduplication coverage for text-level deduplication, corpus quality evaluation, and Simhash/fingerprints.
- Treat broad production web crawling, cross-domain frontier scheduling, anti-bot bypassing, authenticated scraping, and JavaScript rendering as out-of-scope long-tail gaps; Trafilatura provides a docs-supported focused crawler, not a full web-scale crawler.
Evidence basis
This sub-skill is distilled from repository evidence covering download APIs, CLI navigation, feed/sitemap parsing, focused crawling, URL filtering, settings, and offline/native tests. Runtime guidance is self-contained; source paths may appear only as provenance labels in references.
04 同仓库更多 Skills
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
python-api
Programmatic Object-Detection-Metrics API guidance for BoundingBox, BoundingBoxes, Evaluator, enums, coordinate conversion, AP/mAP semantics, and source-style import diagnostics.
file-evaluation
Folder-based PASCAL VOC AP/mAP evaluation from ground-truth and detection text files.
object-detection-metrics
Guides legacy Object-Detection-Metrics PASCAL VOC AP/mAP evaluation from detection text folders and direct Python API classes.
nx-workspace-patterns
Configure and optimize Nx monorepo workspaces. Use when setting up Nx, configuring project boundaries, optimizing build caching, or implementing affected commands.