AGENT SKILL

scraping-scale-reliability-and-data-quality

Use when a scraper has to run reliably and produce trustworthy data: scale and reliability engineering (concurrency, rate limiting, retries and backoff, proxies, scheduling), data quality and validation against the silent breakage that produces plausible-looking wrong data for months, and storage, pipelines and operations including deduplication, change detection and monitoring.

License 许可
MIT
最近更新
2026-11-15
Stars
★ 5
Forks
3

01 前言 Front Matter

name
scraping-scale-reliability-and-data-quality
description
Use when a scraper has to run reliably and produce trustworthy data: scale and reliability engineering (concurrency, rate limiting, retries and backoff, proxies, scheduling), data quality and validation against the silent breakage that produces plausible-looking wrong data for months, and storage, pipelines and operations including deduplication, change detection and monitoring.

03 SKILL.md 正文

Web Scraping: Scale and Reliability, Data Quality, and Storage and Operations

Part 3 of 5 of the Web Scraping reference (plugin web-scraping), covering §8–§10. Sibling skills: scraping-legal-landscape-and-permissions (§0–§3), scraping-tooling-extraction-and-blocking (§4–§7), scraping-ethics-ai-corpora-and-site-defense (§11–§13), scraping-reference (§14–§19). Section numbers are shared across the set; a reference written as §N → skill points into that sibling skill.

Currency: Verified August 2026. See §16 → scraping-reference for the currency snapshot and what goes stale first.

How to read this. Reference, not tutorial. Three markers:

  • [DURABLE] — how HTTP and the web work, engineering practice, or ethics. Doesn't expire.
  • [VERSIONED] — case law, regulation, tooling, platform behaviour. This domain's legal layer is moving faster than its technical layer right now. Verify.
  • [CONTESTED] — genuine disagreement.

⚠️ GOTCHA boxes mark the mistakes that get you blocked, sued, or quietly producing wrong data for months.

This is not legal advice. §1 → scraping-legal-landscape-and-permissions is a map of the frameworks that apply and the questions to put to counsel, not an answer for your jurisdiction or your use case. The legal position depends heavily on what data, from where, about whom, and for what purpose — and those four variables move the answer more than any technical choice you make.

The three framings that organize everything below:

  1. The legal question is not "is scraping legal." It is four separate questions — how did you access it (computer-access law), what did you agree to (contract), what is the data (copyright, personal data), and what will you do with it (privacy, AI training, resale). They have different answers and you need all four (§1 → scraping-legal-landscape-and-permissions).
  2. You are a guest on someone else's infrastructure, and they're paying for it. Nearly every operational rule in this document — rate limiting, caching, conditional requests, off-peak scheduling — follows from taking that seriously, and doing so also happens to be what keeps you unblocked (§11 → scraping-ethics-ai-corpora-and-site-defense).
  3. Scrapers rot. A scraper is a hard dependency on someone else's HTML, which they will change without telling you. Silent breakage that produces plausible-looking wrong data is the characteristic failure of this field (§9), and it's worse than a crash because nobody notices.

§8. Scale and Reliability

[DURABLE] The architecture that survives contact with reality:

URL frontier (queue, deduplicated, prioritized)
   → fetcher pool (rate-limited PER DOMAIN, retries with backoff)
     → response cache (raw HTML stored — see below)
       → parser (pure function: HTML → structured record)
         → validator (§9)
           → storage + change detection

[DURABLE] Store the raw response, not just the parsed output. This is the single most valuable architectural decision in a scraping system: when your parser turns out to have been wrong for three weeks, you can re-parse history instead of re-crawling — which is cheaper for you and, more importantly, costs the target site nothing.

Rate limiting is per-domain, not global. A token bucket per host, with concurrency caps. Scrapy's AutoThrottle adapts to observed latency, which is both polite and effective.

Retries: exponential backoff with jitter, only on transient failures (429, 5xx, timeouts), with a cap. ⚠️ Retrying a 404 or a 403 is just extra load — classify errors before retrying.

Also: conditional requests (If-Modified-Since, If-None-Match → 304 responses cost the server almost nothing), incremental crawling (only fetch what changed — sitemaps with lastmod help), checkpointing so a crash doesn't restart from zero, idempotent writes so a re-run doesn't duplicate, and distributed queues (Redis, SQS) when one machine isn't enough.

Proxies: legitimate uses exist — geographic content variation, avoiding single-IP saturation, and reliability. ⚠️ But note that "rotating residential proxies to evade blocking" is the use case §7.2 → scraping-tooling-extraction-and-blocking flags, and separately, residential proxy networks have documented supply-chain ethics problems around how consumer bandwidth is sourced and whether those users understood what they consented to. That's worth knowing before you buy.


§9. Data Quality

[DURABLE] This is the section that separates a scraper from a data pipeline, and it is the one people skip.

⚠️ GOTCHA — silent breakage is the characteristic failure of this field. A site redesigns. Your selector no longer matches. Your extractor returns empty strings. Your pipeline runs green, your dashboard shows data, and every number is wrong. Nobody notices for weeks because nothing crashed.

A scraper that crashes is a good scraper. A scraper that silently returns None for the price field is a liability.

Validate every record: required fields present; types correct; values within plausible ranges (a price of 0 or 10,000,000 is a parser bug, not a bargain); enums in their expected set; dates sane.

Validate every batch — this is where breakage actually surfaces: record count within expected bounds, null rate per field compared against the historical baseline (the single most effective breakage detector), value distributions not suddenly shifted, duplicate rate stable. Alert on the delta, not on absolute failure.

Cross-check against a second source, an official API, or a manual spot-check on a sample. [DURABLE] Manually verify a handful of records at the start and after every change — it takes ten minutes and catches the class of error nothing else will.

Also track provenance: source URL, fetch timestamp, parser version, and raw-response reference on every record. When you find a problem you need to know exactly which records are affected.

⚠️ And in 2026, add one more check: with tarpit and content-poisoning defences now deployed (§7.1 → scraping-tooling-extraction-and-blocking), verify that what you're collecting is real content, not generated filler served to something the site classified as a bot.


§10. Storage and Operations

Format by purpose: JSON/JSONL for raw and semi-structured; Parquet for analytics (columnar, compressed, and far faster to query); a relational database for relationships and deduplication; object storage for raw HTML archives; a vector store only if you're doing retrieval (§12 → scraping-ethics-ai-corpora-and-site-defense).

[DURABLE] Design for change detection, not just snapshots. Most scraping value comes from what changed — price movements, new listings, removed items. That means either storing versions with valid-from/valid-to, or storing a content hash per record and recording transitions.

Schema evolution: sites add and remove fields, so store the raw payload alongside the parsed record and make your schema additive.

Operational monitoring: success rate by domain, latency, block/CAPTCHA rate, records-per-run against baseline, field-level null rates (§9), cost per record, and queue depth. Alert on trends, not just failures — a slow decline in fields extracted is the signal that matters.

Legal and ethical hygiene at the data layer: know your retention period and enforce it; be able to delete a person's data on request if you hold personal data (GDPR/CCPA rights are not optional and "it's in a training set" is not an answer — §1.6 → scraping-legal-landscape-and-permissions); document provenance and legal basis per source, because that documentation is what a regulator will ask for and it is very hard to reconstruct retroactively.

04 同仓库更多 Skills

the-vibey-project
the-vibey-project/vibey

white-paper-writing

Use when writing white papers, technical reports, policy documents, thought leadership pieces, or any long-form authoritative document. Triggers on requests to write, structure, outline, or review white papers, technical reports, position papers, policy briefs, or evidence-driven arguments. Also use for B2B content marketing strategy, technical documentation requiring authority, academic-style reports, or any document that must educate while persuading. Use when someone asks "how should I structure this" for a long-form document.

★ 5
the-vibey-project
the-vibey-project/vibey

prose-technical-mechanics

Use when editing or writing prose at a technical/mechanical level: choosing between Anglo-Saxon and Latinate diction, placing information in topic vs stress positions, writing periodic vs cumulative sentences, handling sentence rhythm, punctuation as logic, transitions, paragraph structure, or phonesthetics. Use when someone wants to understand WHY prose works or doesn't, not just surface-level style advice. Triggers on word choice, sentence structure, emphasis, rhythm, diction, or any deep writing mechanics question.

★ 5
the-vibey-project
the-vibey-project/vibey

prose-mechanics

Use when writing, editing, or analyzing prose style; when trying to avoid generic AI-sounding writing; when applying specific prose styles such as journalistic, academic, narrative, KJV biblical, Hemingway plain, Latinate/Jamesian, Orwellian, Augustan, Victorian, Early Modern (Bacon, Browne, Milton), or free indirect discourse; or when coaching someone on writing voice and mechanics. Triggers on any request to improve, analyze, or write in a specific prose style, to fix "AI-sounding" writing, to make prose more vivid or concrete, to vary sentence rhythm, or to understand how a particular writer achieves their effects.

★ 5
the-vibey-project
the-vibey-project/vibey

narrative-structure

Use when structuring stories, pitches, presentations, white papers, or any persuasive narrative; analyzing the hero's journey in fiction or non-fiction; applying StoryBrand for business communication; designing narrative arcs for thought leadership; or understanding Vogler, Snyder, Propp, or Campbell frameworks. Triggers on any narrative structure, storytelling, pitch structure, StoryBrand, hero's journey, three-act structure, Freytag, Save the Cat, narrative arc, or "how should I structure this story/pitch/presentation" request.

★ 5
the-vibey-project
the-vibey-project/vibey

legal-brief-writing

Use when writing legal briefs, motions, or any court filing; when analyzing judicial cognition and persuasion; when applying elite brief-writing techniques from Roberts, Clement, Kagan, Garner; or when structuring legal arguments for maximum persuasion. Also use for autistic/neurodivergent advocates who need to understand how their natural communication style maps to elite legal advocacy. Triggers on any legal writing, brief drafting, motion preparation, or legal argument structuring request.

★ 5
the-vibey-project
the-vibey-project/vibey

journal-publishability-criteria

Use whenever a research paper, article or preprint is being written, revised, reviewed or published, and specifically before every publication of docs/paper.md. Covers what legitimate journals actually gate on, in order of gating — the editor's desk screen (scope, novelty, package problems), what reviewers judge (methodological soundness, claims the evidence can carry, statistical and qualitative reporting), EQUATOR reporting guidelines, the ICMJE authorship criteria, COPE's view of text recycling, the near-universal generative-AI policy (no AI author, disclosure, author responsibility), TOP 2025 transparency standards, how to verify a journal is legitimate, submitting without an institution, and a pre-submission checklist. Triggers on requests to evaluate, review, prepare or submit a paper, to judge publishability, to write a declarations or data-availability section, to disclose AI use, or to check a journal.

★ 5