AGENT SKILL
seo-crawl
暂无简介
- License 许可
- MIT
- 所属仓库
- bonc-ai/cogseed
- 最近更新
- 2026-11-13
- Stars
- ★ 19
- Forks
- 10
01 前言 Front Matter
- name
- seo-crawl
- category
- data
- ownerAgent
- e064dca9e1bd
- description_en
- Fetch a URL and extract on-page SEO/GEO fields as structured JSON (title/description/canonical/heading hierarchy/structured data/hreflang/image alt/internal+external links/robots), SSRF-guarded and system-proxy aware; For: 'grab this page's SEO fields', 'pull on-page data for diagnosis'; Triggers: crawl, fetch page, on-page, scrape page, extract fields
- description_zh
- 抓取一个 URL 并抽取 on-page SEO/GEO 字段为结构化 JSON(标题/描述/canonical/标题层级/结构化数据/hreflang/图片 alt/内外链/robots 等),带 SSRF 防护与系统代理支持;适合"抓一下这个页面的 SEO 字段""取回这个站的 on-page 数据做诊断";触发词:抓取、采集、on-page、抓页面、crawl、页面字段
02 资源文件 10
- 参考文档artifact.yaml937 B
- 参考文档evals/evals.json263 B
- 参考文档references/business-ontology.yaml64 B
- 参考文档references/security-manifest.yaml2.8 KB
- 参考文档references/skill-spec.yaml466 B
- 参考文档references/validation-contract.yaml50 B
- 参考文档schemas.json40 B
- 脚本scripts/crawl.py21.4 KB
- 脚本scripts/url_safety.py7.5 KB
- 脚本test/test_crawl.py17.1 KB
03 SKILL.md 正文
seo-crawl
Fetch one URL and return the raw on-page signals the SEO/GEO audits consume. This is the data-acquisition step: it does NOT score or judge — it extracts facts.
When to use
- The diagnose flow needs the on-page facts for a target URL before any audit runs.
- You need the site's
robots.txt+ declared sitemaps alongside the page. - Re-crawling a localhost / preview URL to re-test after an edit (the "apply → re-test" loop).
When NOT to use
- Scoring, bucketing, or producing findings — that is the technical/content/GEO audit step (this skill only extracts).
- Multi-page site crawling at scale — this fetches the single given URL (+ its origin
robots.txt). Breadth-first crawl is a separate concern. - Rendering JavaScript-built DOM — this reads the raw HTML as AI-citation crawlers do; client-rendered content is intentionally out of scope.
Preconditions
- Network access to the target. Honors
HTTP_PROXY/HTTPS_PROXY/NO_PROXY; in fake-ip proxy environments (Clash/Surge) a configured proxy is required because direct DNS returns reserved 198.18.0.0/15 addresses. - Python 3.9+ (stdlib only — no third-party packages).
- Safety: scheme is restricted to http/https; the host is checked against private/loopback/link-local/cloud-metadata ranges and obfuscated-IP forms; on the direct path the connection is pinned to a validated public IP and every redirect hop is re-validated.
How to call
"$COGSEED_NODE" "$COGSEED_PC_DIR/bin/run-skill.cjs" seo-crawl crawl -- <url> [--timeout 20] [--user-agent "<ua>"] [--no-robots]
<url>(required): absolute http(s) URL.--timeoutseconds per request (default 20).--user-agentoverride the crawler UA.--no-robotsskip the site-levelrobots.txtfetch.
Expected output
JSON on stdout. Success:
{ "ok": true, "data": {
"site": { "origin": "https://example.com", "fetched_at": "...Z",
"robots": { "exists": true, "status": 200, "sitemaps": ["https://example.com/sitemap.xml"], "text": "…" } },
"pages": [ {
"url": "...", "final_url": "...", "status_code": 200, "redirect_chain": [],
"response_time_ms": 0, "https": true, "lang": "en", "charset": "utf-8",
"title": "...", "title_length": 0, "meta_description": "...", "meta_robots": null, "canonical": "...",
"og_title": "...", "og_description": "...", "og_image": "...", "twitter_card": "...", "viewport": "...",
"h1s": ["..."], "h1_count": 1, "h2_count": 0, "heading_order": [1,2,2,3],
"word_count": 0, "images_total": 0, "images_missing_alt": 0, "images": [{"src":"...","alt":null}],
"internal_link_count": 0, "external_link_count": 0, "internal_links": ["..."], "external_links": ["..."],
"has_structured_data": true, "structured_data_types": ["Organization"], "structured_data": [ {} ],
"hreflang_tags": [ {"hreflang":"zh","href":"..."} ],
"is_indexable": true, "noindex": false, "first_paragraph": "..."
} ]
} }
Failure: {"ok": false, "error": "<reason>"} on stderr with a non-zero exit (blocked scheme, non-public host / SSRF guard, DNS failure, too many redirects, network timeout).
Notes
is_indexableis derived (HTTP 200 AND nonoindexin meta robots). Canonical-mismatch and header-levelX-Robots-Tagindexability are judged by the technical audit, not here.structured_datais the parsed JSON-LD objects (capped); invalid JSON-LD blocks are skipped, not error.- Link lists are origin-split, de-duplicated, and capped;
#fragment/mailto:/tel:/javascript:are excluded.
04 同仓库更多 Skills
sample-supply-chain
供应链样本:安装钩子 + 拼写伪装 + 源覆盖。
★ 19
sample-prose-only
纯散文安全守则样本:禁止某行为的说明不等于该行为。
★ 19
sample-obfuscated
混淆载荷样本:解码与执行共现。
★ 19
sample-hidden-content
隐藏内容载荷样本:用于回归「不可见字符不做上下文降权」这一不变量。
★ 19
sample-evasion
反规避样本:危险内容只以「不可见字符切断」与「编码后」两种形式出现。
★ 19
sample-agentic
代理运行时风险样本:配置篡改 + 隐蔽指令。
★ 19