server-log-crawler-analyst
Analyze credential-free public website monitoring evidence from server access logs, HTTP fetches, robots/llms/sitemap checks, and exported summaries while protecting raw IP and private query data. Use this skill when the user asks whether crawlers or AI bots reached a site, whether public SEO/LLM discovery files respond, or how to separate observed facts from unknown Search Console, rank, analytics, or citation claims.
- License 许可
- MIT
- 最近更新
- 2026-06-12
- Stars
- ★ 40
- Forks
- 4
01 前言 Front Matter
- name
- server-log-crawler-analyst
- description
- Analyze credential-free public website monitoring evidence from server access logs, HTTP fetches, robots/llms/sitemap checks, and exported summaries while protecting raw IP and private query data. Use this skill when the user asks whether crawlers or AI bots reached a site, whether public SEO/LLM discovery files respond, or how to separate observed facts from unknown Search Console, rank, analytics, or citation claims.
02 资源文件 8
- 参考文档agents/openai.yaml388 B
- 资产文件assets/crawler-monitor-report.template.md732 B
- 参考文档evals.json1.8 KB
- 其他fixtures/sample-access.log575 B
- 参考文档references/data-source-tiers.md2.0 KB
- 参考文档references/monitoring-backlog.md1.4 KB
- 脚本scripts/analyze_access_log.py3.8 KB
- 参考文档validation-report.md1.2 KB
03 SKILL.md 正文
Server Log Crawler Analyst
Use this skill when credential-free monitoring evidence is enough, or when credentialed Search Console, rank tracking, analytics, or assistant citation tools are unavailable.
Read references/data-source-tiers.md before producing a monitoring report.
Owns
- credential-free monitoring baseline;
- server access-log summary with IP privacy protection;
- crawler/user-agent observation reports;
- public HTTP checks for robots, llms.txt, sitemaps, RSS, and representative URLs;
- data-source tiering;
- monitoring limitations and unknowns;
- backlog handoff for Search Console, rank/SERP, analytics, and LLM citation monitoring.
Does Not Own
- credentialed Search Console or analytics API work;
- scraping search engines;
- bypassing bot protections;
- live firewall or WAF changes;
- claims about rankings or assistant citations without direct evidence;
- storing raw IP logs in public artifacts.
Workflow
- Identify available evidence: public URL fetches, access-log snippet, exported summary, or synthetic fixture.
- Label the evidence tier and collection time.
- If logs are provided, summarize requests by path, status, user-agent class, and redacted IP hash.
- Separate known crawler hits from unknown bot-like traffic.
- Check public discovery endpoints when requested.
- Record limitations: no Search Console, no rank data, no assistant citation data unless explicitly provided.
- Produce a monitoring report using assets/crawler-monitor-report.template.md.
- Hand off credentialed or citation tasks to the backlog.
Non-Negotiables
- Do not publish raw IP addresses.
- Do not store credentials, cookies, API keys, or private analytics exports in skill files.
- Do not infer ranking, indexing, or citation success from server logs alone.
- Do not claim a user-agent is verified Google/OpenAI/Claude/Perplexity unless verification method is documented.
- Do not scrape SERPs or assistants against terms of service.
- Do not change firewall, WAF, robots, or server config from this skill.
Safety And Privacy Boundaries
- Treat raw logs as private by default.
- Use aggregated counts and redacted identifiers.
- Keep exact timestamps only when needed for evidence; otherwise aggregate by hour/day.
- If the user asks to publish a report, remove raw IP, query strings, session IDs, auth paths, and admin URLs unless explicitly needed and safe.
- Public reports should say "observed in provided logs" rather than "definitely crawled by" unless bot verification is complete.
Evidence Labels
Observed: directly in log, HTTP response, or exported report.Verified: observed and checked by an accepted verification method.Inferred: likely based on user-agent/path/status, but not verified.Open: no evidence available in this run.
Validation
Validate skill edits with:
python3 <codex-skills-dir>/senior-skill-architect/scripts/lint_production_skill.py ./skills/server-log-crawler-analyst
python3 ./skills/server-log-crawler-analyst/scripts/analyze_access_log.py ./skills/server-log-crawler-analyst/fixtures/sample-access.log --report <tmp-dir>/server-log-crawler-report.json
python3 ./plans/seo-llm-skill-cluster/scripts/lint_skill_cluster.py .
Forward tests live in evals.json.
Output Shape
Return:
- Evidence source and timestamp.
- Data tier and privacy handling.
- Observed crawler/user-agent classes.
- Public endpoint status checks if performed.
- Suspicious or irrelevant bot traffic summary.
- Unknowns and claims refused.
- Monitoring backlog and next evidence needed.
04 同仓库更多 Skills
ux-journey-architect
Map user journeys, onboarding paths, entry points, first actions, trust points, accessibility risks, and SEO/LLM handoffs for public websites without redesigning the visual system. Use this skill when the user asks how new readers, returning users, project reviewers, community members, buyers, or AI agents should move through a site and what each page must make clear before layout implementation.
technical-seo-schema-engineer
Design deterministic technical SEO and schema.org guidance for crawlable public pages. Use this skill when the user asks for title and meta description rules, canonical and hreflang tags, JSON-LD schema, NewsArticle, Article, TechArticle, BreadcrumbList, WebSite, Organization, Person, OpenGraph, Twitter cards, sitemap, news-sitemap, RSS, robots.txt, llms.txt, publisher logo, ImageObject, or metadata templates that must match visible content.
site-growth-orchestrator
Coordinate SEO, LLM-readable site architecture, UX, content, monitoring, security, and external authority work as a task-plan-driven sequence. Use this skill when the user asks to plan, execute, validate, or route multi-skill website growth work; asks what to do next across semantic core, URL structure, internal linking, schema, llms.txt, UX journeys, analytics, citation monitoring, or white-hat link placement; or wants a central architect that preserves evidence, alarms, handoffs, and tests.
seo-regression-validator
Validate SEO and LLM-readability claims against live HTML, local fixtures, and generated artifacts. Use this skill when the user asks to prove whether title, meta description, canonical, hreflang, JSON-LD, NewsArticle.image, OpenGraph, Twitter cards, RSS autodiscovery, robots.txt, llms.txt, sitemap, news-sitemap, crawler access, or performance evidence is actually present, and when subjective audits must become timestamped pass/fail reports.
semantic-core-architect
Build evidence-labeled semantic cores for SEO, LLM-readable architecture, and site information design. Use this skill when the user asks for semantic core, query clusters, search intents, audience segments, entity/topic maps, language or locale priorities, competitor/source evidence, pillar-topic planning, or a structured handoff before URL architecture and internal linking. It must mark unknown search volume, difficulty, and ranking data as unknown instead of inventing metrics.
llm-friendly-site-architect
Design human-visible, machine-readable website layers for AI assistants, RAG, and agentic search. Use this skill when the user asks for llms.txt, LLM discovery plans, visible TLDR or direct-answer blocks, source trails, entity pages, markdown alternates, AI crawler policy, topic-to-URL maps, assistant-readable public content, or LLM-friendly architecture that must preserve canonical SEO strategy without hidden bot-only facts or duplicate story bodies.