[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M72Y1J8K2QKJ9EBKB873Y1MP":3,"skill-same:01M72Y1J8K2QKJ9EBKB873Y1MP":48},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":28,"repositoryHtmlUrl":29,"repositoryHomepageUrl":6,"ownerHtmlUrl":30,"ownerBlogUrl":31,"ownerDescription":32,"updatedAt":33,"install":34,"resources":47},"01M72Y1J8K2QKJ9EBKB873Y1MP","data-scraper-agent",null,"构建一个全自动化的AI驱动数据收集代理，适用于任何公共来源——招聘网站、价格信息、新闻、GitHub、体育赛事等任何内容。按计划进行抓取，使用免费LLM（Gemini Flash）丰富数据，将结果存储在Notion\u002FSheets\u002FSupabase中，并从用户反馈中学习。完全免费在GitHub Actions上运行。适用于用户希望自动监控、收集或跟踪任何公共数据的场景。","SriramGandhiS\u002FSkills","SriramGandhiS","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F211849986?v=4","01M72WQCW12BYX71C6WN584TTA","Sriram S","01M72WQCWTJ2RD0ET1X31AF1RC",1,"MIT","https:\u002F\u002Fgithub.com\u002FSriramGandhiS\u002FSkills\u002Fblob\u002Fmaster\u002Feverything-claude-code\u002Fdocs\u002Fzh-CN\u002Fskills\u002Fdata-scraper-agent\u002FSKILL.md",[],[],"2026-11-10 22:39:42",false,"---\nname: data-scraper-agent\ndescription: 构建一个全自动化的AI驱动数据收集代理，适用于任何公共来源——招聘网站、价格信息、新闻、GitHub、体育赛事等任何内容。按计划进行抓取，使用免费LLM（Gemini Flash）丰富数据，将结果存储在Notion\u002FSheets\u002FSupabase中，并从用户反馈中学习。完全免费在GitHub Actions上运行。适用于用户希望自动监控、收集或跟踪任何公共数据的场景。\norigin: community\n---\n\n# 数据抓取代理\n\n构建一个生产就绪、AI驱动的数据收集代理，适用于任何公共数据源。\n按计划运行，使用免费LLM丰富结果，存储到数据库，并随时间推移不断改进。\n\n**技术栈：Python · Gemini Flash (免费) · GitHub Actions (免费) · Notion \u002F Sheets \u002F Supabase**\n\n## 何时激活\n\n* 用户想要抓取或监控任何公共网站或API\n* 用户说\"构建一个检查...的机器人\"、\"为我监控X\"、\"从...收集数据\"\n* 用户想要跟踪工作、价格、新闻、仓库、体育比分、事件、列表\n* 用户询问如何自动化数据收集而无需支付托管费用\n* 用户想要一个能根据他们的决策随时间推移变得更智能的代理\n\n## 核心概念\n\n### 三层架构\n\n每个数据抓取代理都有三层：\n\n```\nCOLLECT → ENRICH → STORE\n  │           │        │\nScraper    AI (LLM)  Database\nruns on    scores\u002F   Notion \u002F\nschedule   summarises Sheets \u002F\n           & classifies Supabase\n```\n\n### 免费技术栈\n\n| 层级 | 工具 | 原因 |\n|---|---|---|\n| **抓取** | `requests` + `BeautifulSoup` | 无成本，覆盖80%的公共网站 |\n| **JS渲染的网站** | `playwright` (免费) | 当HTML抓取失败时使用 |\n| **AI丰富** | 通过REST API的Gemini Flash | 500次请求\u002F天，100万令牌\u002F天 — 免费 |\n| **存储** | Notion API | 免费层级，用于审查的优秀UI |\n| **调度** | GitHub Actions cron | 对公共仓库免费 |\n| **学习** | 仓库中的JSON反馈文件 | 零基础设施，在git中持久化 |\n\n### AI模型后备链\n\n构建代理以在配额耗尽时自动在Gemini模型间回退：\n\n```\ngemini-2.0-flash-lite (30 RPM) →\ngemini-2.0-flash (15 RPM) →\ngemini-2.5-flash (10 RPM) →\ngemini-flash-lite-latest (fallback)\n```\n\n### 批量API调用以提高效率\n\n切勿为每个项目单独调用LLM。始终批量处理：\n\n```python\n# BAD: 33 API calls for 33 items\nfor item in items:\n    result = call_ai(item)  # 33 calls → hits rate limit\n\n# GOOD: 7 API calls for 33 items (batch size 5)\nfor batch in chunks(items, size=5):\n    results = call_ai(batch)  # 7 calls → stays within free tier\n```\n\n***\n\n## 工作流程\n\n### 步骤 1: 理解目标\n\n询问用户：\n\n1. **收集什么：** \"数据源是什么？URL \u002F API \u002F RSS \u002F 公共端点？\"\n2. **提取什么：** \"哪些字段重要？标题、价格、URL、日期、分数？\"\n3. **如何存储：** \"结果应该存储在哪里？Notion、Google Sheets、Supabase，还是本地文件？\"\n4. **如何丰富：** \"您希望AI对每个项目进行评分、总结、分类或匹配吗？\"\n5. **频率：** \"应该多久运行一次？每小时、每天、每周？\"\n\n常见的提示示例：\n\n* 招聘网站 → 根据简历评分相关性\n* 产品价格 → 降价时发出警报\n* GitHub仓库 → 总结新版本\n* 新闻源 → 按主题+情感分类\n* 体育结果 → 提取统计数据到跟踪器\n* 活动日历 → 按兴趣筛选\n\n***\n\n### 步骤 2: 设计代理架构\n\n为用户生成以下目录结构：\n\n```\nmy-agent\u002F\n├── config.yaml              # 用户自定义此文件（关键词、过滤器、偏好设置）\n├── profile\u002F\n│   └── context.md           # AI 使用的用户上下文（简历、兴趣、标准）\n├── scraper\u002F\n│   ├── __init__.py\n│   ├── main.py              # 协调器：抓取 → 丰富 → 存储\n│   ├── filters.py           # 基于规则的预过滤器（快速，在 AI 处理之前）\n│   └── sources\u002F\n│       ├── __init__.py\n│       └── source_name.py   # 每个数据源一个文件\n├── ai\u002F\n│   ├── __init__.py\n│   ├── client.py            # Gemini REST 客户端，带模型回退\n│   ├── pipeline.py          # 批量 AI 分析\n│   ├── jd_fetcher.py        # 从 URL 获取完整内容（可选）\n│   └── memory.py            # 从用户反馈中学习\n├── storage\u002F\n│   ├── __init__.py\n│   └── notion_sync.py       # 或 sheets_sync.py \u002F supabase_sync.py\n├── data\u002F\n│   └── feedback.json        # 用户决策历史（自动更新）\n├── .env.example\n├── setup.py                 # 一次性数据库\u002F模式创建\n├── enrich_existing.py       # 对旧行进行 AI 分数回填\n├── requirements.txt\n└── .github\u002F\n    └── workflows\u002F\n        └── scraper.yml      # GitHub Actions 计划任务\n```\n\n***\n\n### 步骤 3: 构建抓取器源\n\n适用于任何数据源的模板：\n\n```python\n# scraper\u002Fsources\u002Fmy_source.py\n\"\"\"\n[Source Name] — scrapes [what] from [where].\nMethod: [REST API \u002F HTML scraping \u002F RSS feed]\n\"\"\"\nimport requests\nfrom bs4 import BeautifulSoup\nfrom datetime import datetime, timezone\nfrom scraper.filters import is_relevant\n\nHEADERS = {\n    \"User-Agent\": \"Mozilla\u002F5.0 (compatible; research-bot\u002F1.0)\",\n}\n\ndef fetch() -> list[dict]:\n    \"\"\"\n    Returns a list of items with consistent schema.\n    Each item must have at minimum: name, url, date_found.\n    \"\"\"\n    results = []\n\n    # ---- REST API source ----\n    resp = requests.get(\"https:\u002F\u002Fapi.example.com\u002Fitems\", headers=HEADERS, timeout=15)\n    if resp.status_code == 200:\n        for item in resp.json().get(\"results\", []):\n            if not is_relevant(item.get(\"title\", \"\")):\n                continue\n            results.append(_normalise(item))\n\n    return results\n\ndef _normalise(raw: dict) -> dict:\n    \"\"\"Convert raw API\u002FHTML data to the standard schema.\"\"\"\n    return {\n        \"name\": raw.get(\"title\", \"\"),\n        \"url\": raw.get(\"link\", \"\"),\n        \"source\": \"MySource\",\n        \"date_found\": datetime.now(timezone.utc).date().isoformat(),\n        # add domain-specific fields here\n    }\n```\n\n**HTML抓取模式：**\n\n```python\nsoup = BeautifulSoup(resp.text, \"lxml\")\nfor card in soup.select(\"[class*='listing']\"):\n    title = card.select_one(\"h2, h3\").get_text(strip=True)\n    link = card.select_one(\"a\")[\"href\"]\n    if not link.startswith(\"http\"):\n        link = f\"https:\u002F\u002Fexample.com{link}\"\n```\n\n**RSS源模式：**\n\n```python\nimport xml.etree.ElementTree as ET\nroot = ET.fromstring(resp.text)\nfor item in root.findall(\".\u002F\u002Fitem\"):\n    title = item.findtext(\"title\", \"\")\n    link = item.findtext(\"link\", \"\")\n```\n\n***\n\n### 步骤 4: 构建Gemini AI客户端\n\n````python\n# ai\u002Fclient.py\nimport os, json, time, requests\n\n_last_call = 0.0\n\nMODEL_FALLBACK = [\n    \"gemini-2.0-flash-lite\",\n    \"gemini-2.0-flash\",\n    \"gemini-2.5-flash\",\n    \"gemini-flash-lite-latest\",\n]\n\ndef generate(prompt: str, model: str = \"\", rate_limit: float = 7.0) -> dict:\n    \"\"\"Call Gemini with auto-fallback on 429. Returns parsed JSON or {}.\"\"\"\n    global _last_call\n\n    api_key = os.environ.get(\"GEMINI_API_KEY\", \"\")\n    if not api_key:\n        return {}\n\n    elapsed = time.time() - _last_call\n    if elapsed \u003C rate_limit:\n        time.sleep(rate_limit - elapsed)\n\n    models = [model] + [m for m in MODEL_FALLBACK if m != model] if model else MODEL_FALLBACK\n    _last_call = time.time()\n\n    for m in models:\n        url = f\"https:\u002F\u002Fgenerativelanguage.googleapis.com\u002Fv1beta\u002Fmodels\u002F{m}:generateContent?key={api_key}\"\n        payload = {\n            \"contents\": [{\"parts\": [{\"text\": prompt}]}],\n            \"generationConfig\": {\n                \"responseMimeType\": \"application\u002Fjson\",\n                \"temperature\": 0.3,\n                \"maxOutputTokens\": 2048,\n            },\n        }\n        try:\n            resp = requests.post(url, json=payload, timeout=30)\n            if resp.status_code == 200:\n                return _parse(resp)\n            if resp.status_code in (429, 404):\n                time.sleep(1)\n                continue\n            return {}\n        except requests.RequestException:\n            return {}\n\n    return {}\n\ndef _parse(resp) -> dict:\n    try:\n        text = (\n            resp.json()\n            .get(\"candidates\", [{}])[0]\n            .get(\"content\", {})\n            .get(\"parts\", [{}])[0]\n            .get(\"text\", \"\")\n            .strip()\n        )\n        if text.startswith(\"```\"):\n            text = text.split(\"\\n\", 1)[-1].rsplit(\"```\", 1)[0]\n        return json.loads(text)\n    except (json.JSONDecodeError, KeyError):\n        return {}\n````\n\n***\n\n### 步骤 5: 构建AI管道（批量）\n\n```python\n# ai\u002Fpipeline.py\nimport json\nimport yaml\nfrom pathlib import Path\nfrom ai.client import generate\n\ndef analyse_batch(items: list[dict], context: str = \"\", preference_prompt: str = \"\") -> list[dict]:\n    \"\"\"Analyse items in batches. Returns items enriched with AI fields.\"\"\"\n    config = yaml.safe_load((Path(__file__).parent.parent \u002F \"config.yaml\").read_text())\n    model = config.get(\"ai\", {}).get(\"model\", \"gemini-2.5-flash\")\n    rate_limit = config.get(\"ai\", {}).get(\"rate_limit_seconds\", 7.0)\n    min_score = config.get(\"ai\", {}).get(\"min_score\", 0)\n    batch_size = config.get(\"ai\", {}).get(\"batch_size\", 5)\n\n    batches = [items[i:i + batch_size] for i in range(0, len(items), batch_size)]\n    print(f\"  [AI] {len(items)} items → {len(batches)} API calls\")\n\n    enriched = []\n    for i, batch in enumerate(batches):\n        print(f\"  [AI] Batch {i + 1}\u002F{len(batches)}...\")\n        prompt = _build_prompt(batch, context, preference_prompt, config)\n        result = generate(prompt, model=model, rate_limit=rate_limit)\n\n        analyses = result.get(\"analyses\", [])\n        for j, item in enumerate(batch):\n            ai = analyses[j] if j \u003C len(analyses) else {}\n            if ai:\n                score = max(0, min(100, int(ai.get(\"score\", 0))))\n                if min_score and score \u003C min_score:\n                    continue\n                enriched.append({**item, \"ai_score\": score, \"ai_summary\": ai.get(\"summary\", \"\"), \"ai_notes\": ai.get(\"notes\", \"\")})\n            else:\n                enriched.append(item)\n\n    return enriched\n\ndef _build_prompt(batch, context, preference_prompt, config):\n    priorities = config.get(\"priorities\", [])\n    items_text = \"\\n\\n\".join(\n        f\"Item {i+1}: {json.dumps({k: v for k, v in item.items() if not k.startswith('_')})}\"\n        for i, item in enumerate(batch)\n    )\n\n    return f\"\"\"Analyse these {len(batch)} items and return a JSON object.\n\n# Items\n{items_text}\n\n# User Context\n{context[:800] if context else \"Not provided\"}\n\n# User Priorities\n{chr(10).join(f\"- {p}\" for p in priorities)}\n\n{preference_prompt}\n\n# Instructions\nReturn: {{\"analyses\": [{{\"score\": \u003C0-100>, \"summary\": \"\u003C2 sentences>\", \"notes\": \"\u003Cwhy this matches or doesn't>\"}} for each item in order]}}\nBe concise. Score 90+=excellent match, 70-89=good, 50-69=ok, \u003C50=weak.\"\"\"\n```\n\n***\n\n### 步骤 6: 构建反馈学习系统\n\n```python\n# ai\u002Fmemory.py\n\"\"\"Learn from user decisions to improve future scoring.\"\"\"\nimport json\nfrom pathlib import Path\n\nFEEDBACK_PATH = Path(__file__).parent.parent \u002F \"data\" \u002F \"feedback.json\"\n\ndef load_feedback() -> dict:\n    if FEEDBACK_PATH.exists():\n        try:\n            return json.loads(FEEDBACK_PATH.read_text())\n        except (json.JSONDecodeError, OSError):\n            pass\n    return {\"positive\": [], \"negative\": []}\n\ndef save_feedback(fb: dict):\n    FEEDBACK_PATH.parent.mkdir(parents=True, exist_ok=True)\n    FEEDBACK_PATH.write_text(json.dumps(fb, indent=2))\n\ndef build_preference_prompt(feedback: dict, max_examples: int = 15) -> str:\n    \"\"\"Convert feedback history into a prompt bias section.\"\"\"\n    lines = []\n    if feedback.get(\"positive\"):\n        lines.append(\"# Items the user LIKED (positive signal):\")\n        for e in feedback[\"positive\"][-max_examples:]:\n            lines.append(f\"- {e}\")\n    if feedback.get(\"negative\"):\n        lines.append(\"\\n# Items the user SKIPPED\u002FREJECTED (negative signal):\")\n        for e in feedback[\"negative\"][-max_examples:]:\n            lines.append(f\"- {e}\")\n    if lines:\n        lines.append(\"\\nUse these patterns to bias scoring on new items.\")\n    return \"\\n\".join(lines)\n```\n\n**与存储层集成：** 每次运行后，从数据库中查询具有正面\u002F负面状态的项，并使用提取的模式调用 `save_feedback()`。\n\n***\n\n### 步骤 7: 构建存储（Notion示例）\n\n```python\n# storage\u002Fnotion_sync.py\nimport os\nfrom notion_client import Client\nfrom notion_client.errors import APIResponseError\n\n_client = None\n\ndef get_client():\n    global _client\n    if _client is None:\n        _client = Client(auth=os.environ[\"NOTION_TOKEN\"])\n    return _client\n\ndef get_existing_urls(db_id: str) -> set[str]:\n    \"\"\"Fetch all URLs already stored — used for deduplication.\"\"\"\n    client, seen, cursor = get_client(), set(), None\n    while True:\n        resp = client.databases.query(database_id=db_id, page_size=100, **{\"start_cursor\": cursor} if cursor else {})\n        for page in resp[\"results\"]:\n            url = page[\"properties\"].get(\"URL\", {}).get(\"url\", \"\")\n            if url: seen.add(url)\n        if not resp[\"has_more\"]: break\n        cursor = resp[\"next_cursor\"]\n    return seen\n\ndef push_item(db_id: str, item: dict) -> bool:\n    \"\"\"Push one item to Notion. Returns True on success.\"\"\"\n    props = {\n        \"Name\": {\"title\": [{\"text\": {\"content\": item.get(\"name\", \"\")[:100]}}]},\n        \"URL\": {\"url\": item.get(\"url\")},\n        \"Source\": {\"select\": {\"name\": item.get(\"source\", \"Unknown\")}},\n        \"Date Found\": {\"date\": {\"start\": item.get(\"date_found\")}},\n        \"Status\": {\"select\": {\"name\": \"New\"}},\n    }\n    # AI fields\n    if item.get(\"ai_score\") is not None:\n        props[\"AI Score\"] = {\"number\": item[\"ai_score\"]}\n    if item.get(\"ai_summary\"):\n        props[\"Summary\"] = {\"rich_text\": [{\"text\": {\"content\": item[\"ai_summary\"][:2000]}}]}\n    if item.get(\"ai_notes\"):\n        props[\"Notes\"] = {\"rich_text\": [{\"text\": {\"content\": item[\"ai_notes\"][:2000]}}]}\n\n    try:\n        get_client().pages.create(parent={\"database_id\": db_id}, properties=props)\n        return True\n    except APIResponseError as e:\n        print(f\"[notion] Push failed: {e}\")\n        return False\n\ndef sync(db_id: str, items: list[dict]) -> tuple[int, int]:\n    existing = get_existing_urls(db_id)\n    added = skipped = 0\n    for item in items:\n        if item.get(\"url\") in existing:\n            skipped += 1; continue\n        if push_item(db_id, item):\n            added += 1; existing.add(item[\"url\"])\n        else:\n            skipped += 1\n    return added, skipped\n```\n\n***\n\n### 步骤 8: 在 main.py 中编排\n\n```python\n# scraper\u002Fmain.py\nimport os, sys, yaml\nfrom pathlib import Path\nfrom dotenv import load_dotenv\n\nload_dotenv()\n\nfrom scraper.sources import my_source          # add your sources\n\n# NOTE: This example uses Notion. If storage.provider is \"sheets\" or \"supabase\",\n# replace this import with storage.sheets_sync or storage.supabase_sync and update\n# the env var and sync() call accordingly.\nfrom storage.notion_sync import sync\n\nSOURCES = [\n    (\"My Source\", my_source.fetch),\n]\n\ndef ai_enabled():\n    return bool(os.environ.get(\"GEMINI_API_KEY\"))\n\ndef main():\n    config = yaml.safe_load((Path(__file__).parent.parent \u002F \"config.yaml\").read_text())\n    provider = config.get(\"storage\", {}).get(\"provider\", \"notion\")\n\n    # Resolve the storage target identifier from env based on provider\n    if provider == \"notion\":\n        db_id = os.environ.get(\"NOTION_DATABASE_ID\")\n        if not db_id:\n            print(\"ERROR: NOTION_DATABASE_ID not set\"); sys.exit(1)\n    else:\n        # Extend here for sheets (SHEET_ID) or supabase (SUPABASE_TABLE) etc.\n        print(f\"ERROR: provider '{provider}' not yet wired in main.py\"); sys.exit(1)\n\n    config = yaml.safe_load((Path(__file__).parent.parent \u002F \"config.yaml\").read_text())\n    all_items = []\n\n    for name, fetch_fn in SOURCES:\n        try:\n            items = fetch_fn()\n            print(f\"[{name}] {len(items)} items\")\n            all_items.extend(items)\n        except Exception as e:\n            print(f\"[{name}] FAILED: {e}\")\n\n    # Deduplicate by URL\n    seen, deduped = set(), []\n    for item in all_items:\n        if (url := item.get(\"url\", \"\")) and url not in seen:\n            seen.add(url); deduped.append(item)\n\n    print(f\"Unique items: {len(deduped)}\")\n\n    if ai_enabled() and deduped:\n        from ai.memory import load_feedback, build_preference_prompt\n        from ai.pipeline import analyse_batch\n\n        # load_feedback() reads data\u002Ffeedback.json written by your feedback sync script.\n        # To keep it current, implement a separate feedback_sync.py that queries your\n        # storage provider for items with positive\u002Fnegative statuses and calls save_feedback().\n        feedback = load_feedback()\n        preference = build_preference_prompt(feedback)\n        context_path = Path(__file__).parent.parent \u002F \"profile\" \u002F \"context.md\"\n        context = context_path.read_text() if context_path.exists() else \"\"\n        deduped = analyse_batch(deduped, context=context, preference_prompt=preference)\n    else:\n        print(\"[AI] Skipped — GEMINI_API_KEY not set\")\n\n    added, skipped = sync(db_id, deduped)\n    print(f\"Done — {added} new, {skipped} existing\")\n\nif __name__ == \"__main__\":\n    main()\n```\n\n***\n\n### 步骤 9: GitHub Actions工作流\n\n```yaml\n# .github\u002Fworkflows\u002Fscraper.yml\nname: Data Scraper Agent\n\non:\n  schedule:\n    - cron: \"0 *\u002F3 * * *\"  # every 3 hours — adjust to your needs\n  workflow_dispatch:        # allow manual trigger\n\npermissions:\n  contents: write   # required for the feedback-history commit step\n\njobs:\n  scrape:\n    runs-on: ubuntu-latest\n    timeout-minutes: 20\n\n    steps:\n      - uses: actions\u002Fcheckout@v4\n\n      - uses: actions\u002Fsetup-python@v5\n        with:\n          python-version: \"3.11\"\n          cache: \"pip\"\n\n      - run: pip install -r requirements.txt\n\n      # Uncomment if Playwright is enabled in requirements.txt\n      # - name: Install Playwright browsers\n      #   run: python -m playwright install chromium --with-deps\n\n      - name: Run agent\n        env:\n          NOTION_TOKEN: ${{ secrets.NOTION_TOKEN }}\n          NOTION_DATABASE_ID: ${{ secrets.NOTION_DATABASE_ID }}\n          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}\n        run: python -m scraper.main\n\n      - name: Commit feedback history\n        run: |\n          git config user.name \"github-actions[bot]\"\n          git config user.email \"github-actions[bot]@users.noreply.github.com\"\n          git add data\u002Ffeedback.json || true\n          git diff --cached --quiet || git commit -m \"chore: update feedback history\"\n          git push\n```\n\n***\n\n### 步骤 10: config.yaml 模板\n\n```yaml\n# Customise this file — no code changes needed\n\n# What to collect (pre-filter before AI)\nfilters:\n  required_keywords: []      # item must contain at least one\n  blocked_keywords: []       # item must not contain any\n\n# Your priorities — AI uses these for scoring\npriorities:\n  - \"example priority 1\"\n  - \"example priority 2\"\n\n# Storage\nstorage:\n  provider: \"notion\"         # notion | sheets | supabase | sqlite\n\n# Feedback learning\nfeedback:\n  positive_statuses: [\"Saved\", \"Applied\", \"Interested\"]\n  negative_statuses: [\"Skip\", \"Rejected\", \"Not relevant\"]\n\n# AI settings\nai:\n  enabled: true\n  model: \"gemini-2.5-flash\"\n  min_score: 0               # filter out items below this score\n  rate_limit_seconds: 7      # seconds between API calls\n  batch_size: 5              # items per API call\n```\n\n***\n\n## 常见抓取模式\n\n### 模式 1: REST API（最简单）\n\n```python\nresp = requests.get(url, params={\"q\": query}, headers=HEADERS, timeout=15)\nitems = resp.json().get(\"results\", [])\n```\n\n### 模式 2: HTML抓取\n\n```python\nsoup = BeautifulSoup(resp.text, \"lxml\")\nfor card in soup.select(\".listing-card\"):\n    title = card.select_one(\"h2\").get_text(strip=True)\n    href = card.select_one(\"a\")[\"href\"]\n```\n\n### 模式 3: RSS源\n\n```python\nimport xml.etree.ElementTree as ET\nroot = ET.fromstring(resp.text)\nfor item in root.findall(\".\u002F\u002Fitem\"):\n    title = item.findtext(\"title\", \"\")\n    link = item.findtext(\"link\", \"\")\n    pub_date = item.findtext(\"pubDate\", \"\")\n```\n\n### 模式 4: 分页API\n\n```python\npage = 1\nwhile True:\n    resp = requests.get(url, params={\"page\": page, \"limit\": 50}, timeout=15)\n    data = resp.json()\n    items = data.get(\"results\", [])\n    if not items:\n        break\n    for item in items:\n        results.append(_normalise(item))\n    if not data.get(\"has_more\"):\n        break\n    page += 1\n```\n\n### 模式 5: JS渲染页面（Playwright）\n\n```python\nfrom playwright.sync_api import sync_playwright\n\nwith sync_playwright() as p:\n    browser = p.chromium.launch()\n    page = browser.new_page()\n    page.goto(url)\n    page.wait_for_selector(\".listing\")\n    html = page.content()\n    browser.close()\n\nsoup = BeautifulSoup(html, \"lxml\")\n```\n\n***\n\n## 需要避免的反模式\n\n| 反模式 | 问题 | 修复方法 |\n|---|---|---|\n| 每个项目调用一次LLM | 立即达到速率限制 | 每次调用批量处理5个项目 |\n| 代码中硬编码关键字 | 不可重用 | 将所有配置移动到 `config.yaml` |\n| 没有速率限制的抓取 | IP被禁止 | 在请求之间添加 `time.sleep(1)` |\n| 在代码中存储密钥 | 安全风险 | 始终使用 `.env` + GitHub Secrets |\n| 没有去重 | 重复行堆积 | 在推送前始终检查URL |\n| 忽略 `robots.txt` | 法律\u002F道德风险 | 遵守爬虫规则；尽可能使用公共API |\n| 使用 `requests` 处理JS渲染的网站 | 空响应 | 使用Playwright或查找底层API |\n| `maxOutputTokens` 太低 | JSON截断，解析错误 | 对批量响应使用2048+ |\n\n***\n\n## 免费层级限制参考\n\n| 服务 | 免费限制 | 典型用法 |\n|---|---|---|\n| Gemini Flash Lite | 30 RPM, 1500 RPD | 以3小时间隔约56次请求\u002F天 |\n| Gemini 2.0 Flash | 15 RPM, 1500 RPD | 良好的后备选项 |\n| Gemini 2.5 Flash | 10 RPM, 500 RPD | 谨慎使用 |\n| GitHub Actions | 无限（公共仓库） | 约20分钟\u002F天 |\n| Notion API | 无限 | 约200次写入\u002F天 |\n| Supabase | 500MB DB, 2GB传输 | 适用于大多数代理 |\n| Google Sheets API | 300次请求\u002F分钟 | 适用于小型代理 |\n\n***\n\n## 需求模板\n\n```\nrequests==2.31.0\nbeautifulsoup4==4.12.3\nlxml==5.1.0\npython-dotenv==1.0.1\npyyaml==6.0.2\nnotion-client==2.2.1   # 如需使用 Notion\n# playwright==1.40.0   # 针对 JS 渲染的站点，请取消注释\n```\n\n***\n\n## 质量检查清单\n\n在将代理标记为完成之前：\n\n* \\[ ] `config.yaml` 控制所有面向用户的设置 — 没有硬编码的值\n* \\[ ] `profile\u002Fcontext.md` 保存用于AI匹配的用户特定上下文\n* \\[ ] 在每次存储推送前通过URL进行去重\n* \\[ ] Gemini客户端具有模型后备链（4个模型）\n* \\[ ] 批量大小 ≤ 每个API调用5个项目\n* \\[ ] `maxOutputTokens` ≥ 2048\n* \\[ ] `.env` 在 `.gitignore` 中\n* \\[ ] 提供了用于入门的 `.env.example`\n* \\[ ] `setup.py` 在首次运行时创建数据库模式\n* \\[ ] `enrich_existing.py` 回填旧行的AI分数\n* \\[ ] GitHub Actions工作流在每次运行后提交 `feedback.json`\n* \\[ ] README涵盖：在\u003C5分钟内设置，所需的密钥，自定义\n\n***\n\n## 真实世界示例\n\n```\n\"为我构建一个监控 Hacker News 上 AI 初创公司融资新闻的智能体\"\n\"从 3 家电商网站抓取产品价格并在降价时发出提醒\"\n\"追踪标记有 'llm' 或 'agents' 的新 GitHub 仓库——并为每个仓库生成摘要\"\n\"将 LinkedIn 和 Cutshort 上的首席运营官职位列表收集到 Notion 中\"\n\"监控一个提到我公司的 subreddit 帖子——并进行情感分类\"\n\"每日从 arXiv 抓取我关注主题的新学术论文\"\n\"追踪体育赛事结果并在 Google Sheets 中维护动态更新的表格\"\n\"构建一个房地产房源监控器——在新房源价格低于 1 千万卢比时发出提醒\"\n```\n\n***\n\n## 参考实现\n\n一个使用此确切架构构建的完整工作代理将抓取4+个数据源，\n批量处理Gemini调用，从存储在Notion中的\"已应用\"\u002F\"已拒绝\"决策中学习，并且\n在GitHub Actions上100%免费运行。按照上述步骤1-9构建您自己的代理。\n","# 数据抓取代理\n\n构建一个生产就绪、AI驱动的数据收集代理，适用于任何公共数据源。\n按计划运行，使用免费LLM丰富结果，存储到数据库，并随时间推移不断改进。\n\n**技术栈：Python · Gemini Flash (免费) · GitHub Actions (免费) · Notion \u002F Sheets \u002F Supabase**\n\n## 何时激活\n\n* 用户想要抓取或监控任何公共网站或API\n* 用户说\"构建一个检查...的机器人\"、\"为我监控X\"、\"从...收集数据\"\n* 用户想要跟踪工作、价格、新闻、仓库、体育比分、事件、列表\n* 用户询问如何自动化数据收集而无需支付托管费用\n* 用户想要一个能根据他们的决策随时间推移变得更智能的代理\n\n## 核心概念\n\n### 三层架构\n\n每个数据抓取代理都有三层：\n\n```\nCOLLECT → ENRICH → STORE\n  │           │        │\nScraper    AI (LLM)  Database\nruns on    scores\u002F   Notion \u002F\nschedule   summarises Sheets \u002F\n           & classifies Supabase\n```\n\n### 免费技术栈\n\n| 层级 | 工具 | 原因 |\n|---|---|---|\n| **抓取** | `requests` + `BeautifulSoup` | 无成本，覆盖80%的公共网站 |\n| **JS渲染的网站** | `playwright` (免费) | 当HTML抓取失败时使用 |\n| **AI丰富** | 通过REST API的Gemini Flash | 500次请求\u002F天，100万令牌\u002F天 — 免费 |\n| **存储** | Notion API | 免费层级，用于审查的优秀UI |\n| **调度** | GitHub Actions cron | 对公共仓库免费 |\n| **学习** | 仓库中的JSON反馈文件 | 零基础设施，在git中持久化 |\n\n### AI模型后备链\n\n构建代理以在配额耗尽时自动在Gemini模型间回退：\n\n```\ngemini-2.0-flash-lite (30 RPM) →\ngemini-2.0-flash (15 RPM) →\ngemini-2.5-flash (10 RPM) →\ngemini-flash-lite-latest (fallback)\n```\n\n### 批量API调用以提高效率\n\n切勿为每个项目单独调用LLM。始终批量处理：\n\n```python\n# BAD: 33 API calls for 33 items\nfor item in items:\n    result = call_ai(item)  # 33 calls → hits rate limit\n\n# GOOD: 7 API calls for 33 items (batch size 5)\nfor batch in chunks(items, size=5):\n    results = call_ai(batch)  # 7 calls → stays within free tier\n```\n\n***\n\n## 工作流程\n\n### 步骤 1: 理解目标\n\n询问用户：\n\n1. **收集什么：** \"数据源是什么？URL \u002F API \u002F RSS \u002F 公共端点？\"\n2. **提取什么：** \"哪些字段重要？标题、价格、URL、日期、分数？\"\n3. **如何存储：** \"结果应该存储在哪里？Notion、Google Sheets、Supabase，还是本地文件？\"\n4. **如何丰富：** \"您希望AI对每个项目进行评分、总结、分类或匹配吗？\"\n5. **频率：** \"应该多久运行一次？每小时、每天、每周？\"\n\n常见的提示示例：\n\n* 招聘网站 → 根据简历评分相关性\n* 产品价格 → 降价时发出警报\n* GitHub仓库 → 总结新版本\n* 新闻源 → 按主题+情感分类\n* 体育结果 → 提取统计数据到跟踪器\n* 活动日历 → 按兴趣筛选\n\n***\n\n### 步骤 2: 设计代理架构\n\n为用户生成以下目录结构：\n\n```\nmy-agent\u002F\n├── config.yaml              # 用户自定义此文件（关键词、过滤器、偏好设置）\n├── profile\u002F\n│   └── context.md           # AI 使用的用户上下文（简历、兴趣、标准）\n├── scraper\u002F\n│   ├── __init__.py\n│   ├── main.py              # 协调器：抓取 → 丰富 → 存储\n│   ├── filters.py           # 基于规则的预过滤器（快速，在 AI 处理之前）\n│   └── sources\u002F\n│       ├── __init__.py\n│       └── source_name.py   # 每个数据源一个文件\n├── ai\u002F\n│   ├── __init__.py\n│   ├── client.py            # Gemini REST 客户端，带模型回退\n│   ├── pipeline.py          # 批量 AI 分析\n│   ├── jd_fetcher.py        # 从 URL 获取完整内容（可选）\n│   └── memory.py            # 从用户反馈中学习\n├── storage\u002F\n│   ├── __init__.py\n│   └── notion_sync.py       # 或 sheets_sync.py \u002F supabase_sync.py\n├── data\u002F\n│   └── feedback.json        # 用户决策历史（自动更新）\n├── .env.example\n├── setup.py                 # 一次性数据库\u002F模式创建\n├── enrich_existing.py       # 对旧行进行 AI 分数回填\n├── requirements.txt\n└── .github\u002F\n    └── workflows\u002F\n        └── scraper.yml      # GitHub Actions 计划任务\n```\n\n***\n\n### 步骤 3: 构建抓取器源\n\n适用于任何数据源的模板：\n\n```python\n# scraper\u002Fsources\u002Fmy_source.py\n\"\"\"\n[Source Name] — scrapes [what] from [where].\nMethod: [REST API \u002F HTML scraping \u002F RSS feed]\n\"\"\"\nimport requests\nfrom bs4 import BeautifulSoup\nfrom datetime import datetime, timezone\nfrom scraper.filters import is_relevant\n\nHEADERS = {\n    \"User-Agent\": \"Mozilla\u002F5.0 (compatible; research-bot\u002F1.0)\",\n}\n\ndef fetch() -> list[dict]:\n    \"\"\"\n    Returns a list of items with consistent schema.\n    Each item must have at minimum: name, url, date_found.\n    \"\"\"\n    results = []\n\n    # ---- REST API source ----\n    resp = requests.get(\"https:\u002F\u002Fapi.example.com\u002Fitems\", headers=HEADERS, timeout=15)\n    if resp.status_code == 200:\n        for item in resp.json().get(\"results\", []):\n            if not is_relevant(item.get(\"title\", \"\")):\n                continue\n            results.append(_normalise(item))\n\n    return results\n\ndef _normalise(raw: dict) -> dict:\n    \"\"\"Convert raw API\u002FHTML data to the standard schema.\"\"\"\n    return {\n        \"name\": raw.get(\"title\", \"\"),\n        \"url\": raw.get(\"link\", \"\"),\n        \"source\": \"MySource\",\n        \"date_found\": datetime.now(timezone.utc).date().isoformat(),\n        # add domain-specific fields here\n    }\n```\n\n**HTML抓取模式：**\n\n```python\nsoup = BeautifulSoup(resp.text, \"lxml\")\nfor card in soup.select(\"[class*='listing']\"):\n    title = card.select_one(\"h2, h3\").get_text(strip=True)\n    link = card.select_one(\"a\")[\"href\"]\n    if not link.startswith(\"http\"):\n        link = f\"https:\u002F\u002Fexample.com{link}\"\n```\n\n**RSS源模式：**\n\n```python\nimport xml.etree.ElementTree as ET\nroot = ET.fromstring(resp.text)\nfor item in root.findall(\".\u002F\u002Fitem\"):\n    title = item.findtext(\"title\", \"\")\n    link = item.findtext(\"link\", \"\")\n```\n\n***\n\n### 步骤 4: 构建Gemini AI客户端\n\n````python\n# ai\u002Fclient.py\nimport os, json, time, requests\n\n_last_call = 0.0\n\nMODEL_FALLBACK = [\n    \"gemini-2.0-flash-lite\",\n    \"gemini-2.0-flash\",\n    \"gemini-2.5-flash\",\n    \"gemini-flash-lite-latest\",\n]\n\ndef generate(prompt: str, model: str = \"\", rate_limit: float = 7.0) -> dict:\n    \"\"\"Call Gemini with auto-fallback on 429. Returns parsed JSON or {}.\"\"\"\n    global _last_call\n\n    api_key = os.environ.get(\"GEMINI_API_KEY\", \"\")\n    if not api_key:\n        return {}\n\n    elapsed = time.time() - _last_call\n    if elapsed \u003C rate_limit:\n        time.sleep(rate_limit - elapsed)\n\n    models = [model] + [m for m in MODEL_FALLBACK if m != model] if model else MODEL_FALLBACK\n    _last_call = time.time()\n\n    for m in models:\n        url = f\"https:\u002F\u002Fgenerativelanguage.googleapis.com\u002Fv1beta\u002Fmodels\u002F{m}:generateContent?key={api_key}\"\n        payload = {\n            \"contents\": [{\"parts\": [{\"text\": prompt}]}],\n            \"generationConfig\": {\n                \"responseMimeType\": \"application\u002Fjson\",\n                \"temperature\": 0.3,\n                \"maxOutputTokens\": 2048,\n            },\n        }\n        try:\n            resp = requests.post(url, json=payload, timeout=30)\n            if resp.status_code == 200:\n                return _parse(resp)\n            if resp.status_code in (429, 404):\n                time.sleep(1)\n                continue\n            return {}\n        except requests.RequestException:\n            return {}\n\n    return {}\n\ndef _parse(resp) -> dict:\n    try:\n        text = (\n            resp.json()\n            .get(\"candidates\", [{}])[0]\n            .get(\"content\", {})\n            .get(\"parts\", [{}])[0]\n            .get(\"text\", \"\")\n            .strip()\n        )\n        if text.startswith(\"```\"):\n            text = text.split(\"\\n\", 1)[-1].rsplit(\"```\", 1)[0]\n        return json.loads(text)\n    except (json.JSONDecodeError, KeyError):\n        return {}\n````\n\n***\n\n### 步骤 5: 构建AI管道（批量）\n\n```python\n# ai\u002Fpipeline.py\nimport json\nimport yaml\nfrom pathlib import Path\nfrom ai.client import generate\n\ndef analyse_batch(items: list[dict], context: str = \"\", preference_prompt: str = \"\") -> list[dict]:\n    \"\"\"Analyse items in batches. Returns items enriched with AI fields.\"\"\"\n    config = yaml.safe_load((Path(__file__).parent.parent \u002F \"config.yaml\").read_text())\n    model = config.get(\"ai\", {}).get(\"model\", \"gemini-2.5-flash\")\n    rate_limit = config.get(\"ai\", {}).get(\"rate_limit_seconds\", 7.0)\n    min_score = config.get(\"ai\", {}).get(\"min_score\", 0)\n    batch_size = config.get(\"ai\", {}).get(\"batch_size\", 5)\n\n    batches = [items[i:i + batch_size] for i in range(0, len(items), batch_size)]\n    print(f\"  [AI] {len(items)} items → {len(batches)} API calls\")\n\n    enriched = []\n    for i, batch in enumerate(batches):\n        print(f\"  [AI] Batch {i + 1}\u002F{len(batches)}...\")\n        prompt = _build_prompt(batch, context, preference_prompt, config)\n        result = generate(prompt, model=model, rate_limit=rate_limit)\n\n        analyses = result.get(\"analyses\", [])\n        for j, item in enumerate(batch):\n            ai = analyses[j] if j \u003C len(analyses) else {}\n            if ai:\n                score = max(0, min(100, int(ai.get(\"score\", 0))))\n                if min_score and score \u003C min_score:\n                    continue\n                enriched.append({**item, \"ai_score\": score, \"ai_summary\": ai.get(\"summary\", \"\"), \"ai_notes\": ai.get(\"notes\", \"\")})\n            else:\n                enriched.append(item)\n\n    return enriched\n\ndef _build_prompt(batch, context, preference_prompt, config):\n    priorities = config.get(\"priorities\", [])\n    items_text = \"\\n\\n\".join(\n        f\"Item {i+1}: {json.dumps({k: v for k, v in item.items() if not k.startswith('_')})}\"\n        for i, item in enumerate(batch)\n    )\n\n    return f\"\"\"Analyse these {len(batch)} items and return a JSON object.\n\n# Items\n{items_text}\n\n# User Context\n{context[:800] if context else \"Not provided\"}\n\n# User Priorities\n{chr(10).join(f\"- {p}\" for p in priorities)}\n\n{preference_prompt}\n\n# Instructions\nReturn: {{\"analyses\": [{{\"score\": \u003C0-100>, \"summary\": \"\u003C2 sentences>\", \"notes\": \"\u003Cwhy this matches or doesn't>\"}} for each item in order]}}\nBe concise. Score 90+=excellent match, 70-89=good, 50-69=ok, \u003C50=weak.\"\"\"\n```\n\n***\n\n### 步骤 6: 构建反馈学习系统\n\n```python\n# ai\u002Fmemory.py\n\"\"\"Learn from user decisions to improve future scoring.\"\"\"\nimport json\nfrom pathlib import Path\n\nFEEDBACK_PATH = Path(__file__).parent.parent \u002F \"data\" \u002F \"feedback.json\"\n\ndef load_feedback() -> dict:\n    if FEEDBACK_PATH.exists():\n        try:\n            return json.loads(FEEDBACK_PATH.read_text())\n        except (json.JSONDecodeError, OSError):\n            pass\n    return {\"positive\": [], \"negative\": []}\n\ndef save_feedback(fb: dict):\n    FEEDBACK_PATH.parent.mkdir(parents=True, exist_ok=True)\n    FEEDBACK_PATH.write_text(json.dumps(fb, indent=2))\n\ndef build_preference_prompt(feedback: dict, max_examples: int = 15) -> str:\n    \"\"\"Convert feedback history into a prompt bias section.\"\"\"\n    lines = []\n    if feedback.get(\"positive\"):\n        lines.append(\"# Items the user LIKED (positive signal):\")\n        for e in feedback[\"positive\"][-max_examples:]:\n            lines.append(f\"- {e}\")\n    if feedback.get(\"negative\"):\n        lines.append(\"\\n# Items the user SKIPPED\u002FREJECTED (negative signal):\")\n        for e in feedback[\"negative\"][-max_examples:]:\n            lines.append(f\"- {e}\")\n    if lines:\n        lines.append(\"\\nUse these patterns to bias scoring on new items.\")\n    return \"\\n\".join(lines)\n```\n\n**与存储层集成：** 每次运行后，从数据库中查询具有正面\u002F负面状态的项，并使用提取的模式调用 `save_feedback()`。\n\n***\n\n### 步骤 7: 构建存储（Notion示例）\n\n```python\n# storage\u002Fnotion_sync.py\nimport os\nfrom notion_client import Client\nfrom notion_client.errors import APIResponseError\n\n_client = None\n\ndef get_client():\n    global _client\n    if _client is None:\n        _client = Client(auth=os.environ[\"NOTION_TOKEN\"])\n    return _client\n\ndef get_existing_urls(db_id: str) -> set[str]:\n    \"\"\"Fetch all URLs already stored — used for deduplication.\"\"\"\n    client, seen, cursor = get_client(), set(), None\n    while True:\n        resp = client.databases.query(database_id=db_id, page_size=100, **{\"start_cursor\": cursor} if cursor else {})\n        for page in resp[\"results\"]:\n            url = page[\"properties\"].get(\"URL\", {}).get(\"url\", \"\")\n            if url: seen.add(url)\n        if not resp[\"has_more\"]: break\n        cursor = resp[\"next_cursor\"]\n    return seen\n\ndef push_item(db_id: str, item: dict) -> bool:\n    \"\"\"Push one item to Notion. Returns True on success.\"\"\"\n    props = {\n        \"Name\": {\"title\": [{\"text\": {\"content\": item.get(\"name\", \"\")[:100]}}]},\n        \"URL\": {\"url\": item.get(\"url\")},\n        \"Source\": {\"select\": {\"name\": item.get(\"source\", \"Unknown\")}},\n        \"Date Found\": {\"date\": {\"start\": item.get(\"date_found\")}},\n        \"Status\": {\"select\": {\"name\": \"New\"}},\n    }\n    # AI fields\n    if item.get(\"ai_score\") is not None:\n        props[\"AI Score\"] = {\"number\": item[\"ai_score\"]}\n    if item.get(\"ai_summary\"):\n        props[\"Summary\"] = {\"rich_text\": [{\"text\": {\"content\": item[\"ai_summary\"][:2000]}}]}\n    if item.get(\"ai_notes\"):\n        props[\"Notes\"] = {\"rich_text\": [{\"text\": {\"content\": item[\"ai_notes\"][:2000]}}]}\n\n    try:\n        get_client().pages.create(parent={\"database_id\": db_id}, properties=props)\n        return True\n    except APIResponseError as e:\n        print(f\"[notion] Push failed: {e}\")\n        return False\n\ndef sync(db_id: str, items: list[dict]) -> tuple[int, int]:\n    existing = get_existing_urls(db_id)\n    added = skipped = 0\n    for item in items:\n        if item.get(\"url\") in existing:\n            skipped += 1; continue\n        if push_item(db_id, item):\n            added += 1; existing.add(item[\"url\"])\n        else:\n            skipped += 1\n    return added, skipped\n```\n\n***\n\n### 步骤 8: 在 main.py 中编排\n\n```python\n# scraper\u002Fmain.py\nimport os, sys, yaml\nfrom pathlib import Path\nfrom dotenv import load_dotenv\n\nload_dotenv()\n\nfrom scraper.sources import my_source          # add your sources\n\n# NOTE: This example uses Notion. If storage.provider is \"sheets\" or \"supabase\",\n# replace this import with storage.sheets_sync or storage.supabase_sync and update\n# the env var and sync() call accordingly.\nfrom storage.notion_sync import sync\n\nSOURCES = [\n    (\"My Source\", my_source.fetch),\n]\n\ndef ai_enabled():\n    return bool(os.environ.get(\"GEMINI_API_KEY\"))\n\ndef main():\n    config = yaml.safe_load((Path(__file__).parent.parent \u002F \"config.yaml\").read_text())\n    provider = config.get(\"storage\", {}).get(\"provider\", \"notion\")\n\n    # Resolve the storage target identifier from env based on provider\n    if provider == \"notion\":\n        db_id = os.environ.get(\"NOTION_DATABASE_ID\")\n        if not db_id:\n            print(\"ERROR: NOTION_DATABASE_ID not set\"); sys.exit(1)\n    else:\n        # Extend here for sheets (SHEET_ID) or supabase (SUPABASE_TABLE) etc.\n        print(f\"ERROR: provider '{provider}' not yet wired in main.py\"); sys.exit(1)\n\n    config = yaml.safe_load((Path(__file__).parent.parent \u002F \"config.yaml\").read_text())\n    all_items = []\n\n    for name, fetch_fn in SOURCES:\n        try:\n            items = fetch_fn()\n            print(f\"[{name}] {len(items)} items\")\n            all_items.extend(items)\n        except Exception as e:\n            print(f\"[{name}] FAILED: {e}\")\n\n    # Deduplicate by URL\n    seen, deduped = set(), []\n    for item in all_items:\n        if (url := item.get(\"url\", \"\")) and url not in seen:\n            seen.add(url); deduped.append(item)\n\n    print(f\"Unique items: {len(deduped)}\")\n\n    if ai_enabled() and deduped:\n        from ai.memory import load_feedback, build_preference_prompt\n        from ai.pipeline import analyse_batch\n\n        # load_feedback() reads data\u002Ffeedback.json written by your feedback sync script.\n        # To keep it current, implement a separate feedback_sync.py that queries your\n        # storage provider for items with positive\u002Fnegative statuses and calls save_feedback().\n        feedback = load_feedback()\n        preference = build_preference_prompt(feedback)\n        context_path = Path(__file__).parent.parent \u002F \"profile\" \u002F \"context.md\"\n        context = context_path.read_text() if context_path.exists() else \"\"\n        deduped = analyse_batch(deduped, context=context, preference_prompt=preference)\n    else:\n        print(\"[AI] Skipped — GEMINI_API_KEY not set\")\n\n    added, skipped = sync(db_id, deduped)\n    print(f\"Done — {added} new, {skipped} existing\")\n\nif __name__ == \"__main__\":\n    main()\n```\n\n***\n\n### 步骤 9: GitHub Actions工作流\n\n```yaml\n# .github\u002Fworkflows\u002Fscraper.yml\nname: Data Scraper Agent\n\non:\n  schedule:\n    - cron: \"0 *\u002F3 * * *\"  # every 3 hours — adjust to your needs\n  workflow_dispatch:        # allow manual trigger\n\npermissions:\n  contents: write   # required for the feedback-history commit step\n\njobs:\n  scrape:\n    runs-on: ubuntu-latest\n    timeout-minutes: 20\n\n    steps:\n      - uses: actions\u002Fcheckout@v4\n\n      - uses: actions\u002Fsetup-python@v5\n        with:\n          python-version: \"3.11\"\n          cache: \"pip\"\n\n      - run: pip install -r requirements.txt\n\n      # Uncomment if Playwright is enabled in requirements.txt\n      # - name: Install Playwright browsers\n      #   run: python -m playwright install chromium --with-deps\n\n      - name: Run agent\n        env:\n          NOTION_TOKEN: ${{ secrets.NOTION_TOKEN }}\n          NOTION_DATABASE_ID: ${{ secrets.NOTION_DATABASE_ID }}\n          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}\n        run: python -m scraper.main\n\n      - name: Commit feedback history\n        run: |\n          git config user.name \"github-actions[bot]\"\n          git config user.email \"github-actions[bot]@users.noreply.github.com\"\n          git add data\u002Ffeedback.json || true\n          git diff --cached --quiet || git commit -m \"chore: update feedback history\"\n          git push\n```\n\n***\n\n### 步骤 10: config.yaml 模板\n\n```yaml\n# Customise this file — no code changes needed\n\n# What to collect (pre-filter before AI)\nfilters:\n  required_keywords: []      # item must contain at least one\n  blocked_keywords: []       # item must not contain any\n\n# Your priorities — AI uses these for scoring\npriorities:\n  - \"example priority 1\"\n  - \"example priority 2\"\n\n# Storage\nstorage:\n  provider: \"notion\"         # notion | sheets | supabase | sqlite\n\n# Feedback learning\nfeedback:\n  positive_statuses: [\"Saved\", \"Applied\", \"Interested\"]\n  negative_statuses: [\"Skip\", \"Rejected\", \"Not relevant\"]\n\n# AI settings\nai:\n  enabled: true\n  model: \"gemini-2.5-flash\"\n  min_score: 0               # filter out items below this score\n  rate_limit_seconds: 7      # seconds between API calls\n  batch_size: 5              # items per API call\n```\n\n***\n\n## 常见抓取模式\n\n### 模式 1: REST API（最简单）\n\n```python\nresp = requests.get(url, params={\"q\": query}, headers=HEADERS, timeout=15)\nitems = resp.json().get(\"results\", [])\n```\n\n### 模式 2: HTML抓取\n\n```python\nsoup = BeautifulSoup(resp.text, \"lxml\")\nfor card in soup.select(\".listing-card\"):\n    title = card.select_one(\"h2\").get_text(strip=True)\n    href = card.select_one(\"a\")[\"href\"]\n```\n\n### 模式 3: RSS源\n\n```python\nimport xml.etree.ElementTree as ET\nroot = ET.fromstring(resp.text)\nfor item in root.findall(\".\u002F\u002Fitem\"):\n    title = item.findtext(\"title\", \"\")\n    link = item.findtext(\"link\", \"\")\n    pub_date = item.findtext(\"pubDate\", \"\")\n```\n\n### 模式 4: 分页API\n\n```python\npage = 1\nwhile True:\n    resp = requests.get(url, params={\"page\": page, \"limit\": 50}, timeout=15)\n    data = resp.json()\n    items = data.get(\"results\", [])\n    if not items:\n        break\n    for item in items:\n        results.append(_normalise(item))\n    if not data.get(\"has_more\"):\n        break\n    page += 1\n```\n\n### 模式 5: JS渲染页面（Playwright）\n\n```python\nfrom playwright.sync_api import sync_playwright\n\nwith sync_playwright() as p:\n    browser = p.chromium.launch()\n    page = browser.new_page()\n    page.goto(url)\n    page.wait_for_selector(\".listing\")\n    html = page.content()\n    browser.close()\n\nsoup = BeautifulSoup(html, \"lxml\")\n```\n\n***\n\n## 需要避免的反模式\n\n| 反模式 | 问题 | 修复方法 |\n|---|---|---|\n| 每个项目调用一次LLM | 立即达到速率限制 | 每次调用批量处理5个项目 |\n| 代码中硬编码关键字 | 不可重用 | 将所有配置移动到 `config.yaml` |\n| 没有速率限制的抓取 | IP被禁止 | 在请求之间添加 `time.sleep(1)` |\n| 在代码中存储密钥 | 安全风险 | 始终使用 `.env` + GitHub Secrets |\n| 没有去重 | 重复行堆积 | 在推送前始终检查URL |\n| 忽略 `robots.txt` | 法律\u002F道德风险 | 遵守爬虫规则；尽可能使用公共API |\n| 使用 `requests` 处理JS渲染的网站 | 空响应 | 使用Playwright或查找底层API |\n| `maxOutputTokens` 太低 | JSON截断，解析错误 | 对批量响应使用2048+ |\n\n***\n\n## 免费层级限制参考\n\n| 服务 | 免费限制 | 典型用法 |\n|---|---|---|\n| Gemini Flash Lite | 30 RPM, 1500 RPD | 以3小时间隔约56次请求\u002F天 |\n| Gemini 2.0 Flash | 15 RPM, 1500 RPD | 良好的后备选项 |\n| Gemini 2.5 Flash | 10 RPM, 500 RPD | 谨慎使用 |\n| GitHub Actions | 无限（公共仓库） | 约20分钟\u002F天 |\n| Notion API | 无限 | 约200次写入\u002F天 |\n| Supabase | 500MB DB, 2GB传输 | 适用于大多数代理 |\n| Google Sheets API | 300次请求\u002F分钟 | 适用于小型代理 |\n\n***\n\n## 需求模板\n\n```\nrequests==2.31.0\nbeautifulsoup4==4.12.3\nlxml==5.1.0\npython-dotenv==1.0.1\npyyaml==6.0.2\nnotion-client==2.2.1   # 如需使用 Notion\n# playwright==1.40.0   # 针对 JS 渲染的站点，请取消注释\n```\n\n***\n\n## 质量检查清单\n\n在将代理标记为完成之前：\n\n* \\[ ] `config.yaml` 控制所有面向用户的设置 — 没有硬编码的值\n* \\[ ] `profile\u002Fcontext.md` 保存用于AI匹配的用户特定上下文\n* \\[ ] 在每次存储推送前通过URL进行去重\n* \\[ ] Gemini客户端具有模型后备链（4个模型）\n* \\[ ] 批量大小 ≤ 每个API调用5个项目\n* \\[ ] `maxOutputTokens` ≥ 2048\n* \\[ ] `.env` 在 `.gitignore` 中\n* \\[ ] 提供了用于入门的 `.env.example`\n* \\[ ] `setup.py` 在首次运行时创建数据库模式\n* \\[ ] `enrich_existing.py` 回填旧行的AI分数\n* \\[ ] GitHub Actions工作流在每次运行后提交 `feedback.json`\n* \\[ ] README涵盖：在\u003C5分钟内设置，所需的密钥，自定义\n\n***\n\n## 真实世界示例\n\n```\n\"为我构建一个监控 Hacker News 上 AI 初创公司融资新闻的智能体\"\n\"从 3 家电商网站抓取产品价格并在降价时发出提醒\"\n\"追踪标记有 'llm' 或 'agents' 的新 GitHub 仓库——并为每个仓库生成摘要\"\n\"将 LinkedIn 和 Cutshort 上的首席运营官职位列表收集到 Notion 中\"\n\"监控一个提到我公司的 subreddit 帖子——并进行情感分类\"\n\"每日从 arXiv 抓取我关注主题的新学术论文\"\n\"追踪体育赛事结果并在 Google Sheets 中维护动态更新的表格\"\n\"构建一个房地产房源监控器——在新房源价格低于 1 千万卢比时发出提醒\"\n```\n\n***\n\n## 参考实现\n\n一个使用此确切架构构建的完整工作代理将抓取4+个数据源，\n批量处理Gemini调用，从存储在Notion中的\"已应用\"\u002F\"已拒绝\"决策中学习，并且\n在GitHub Actions上100%免费运行。按照上述步骤1-9构建您自己的代理。\n",{"name":5,"origin":24,"description":7},"community","master","d6ca01da0f24a3b0e26e4a8b7b66cbd25d429f5d","everything-claude-code\u002Fdocs\u002Fzh-CN\u002Fskills\u002Fdata-scraper-agent",0,"https:\u002F\u002Fgithub.com\u002FSriramGandhiS\u002FSkills","https:\u002F\u002Fgithub.com\u002FSriramGandhiS","https:\u002F\u002Fsriram.website\u002F","Curious guy who keeps tinkering until something works","2026-05-20 16:21:34",{"source":8,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add SriramGandhiS\u002FSkills --skill data-scraper-agent",{"tool":40,"command":41},"bunx","bunx skills add SriramGandhiS\u002FSkills --skill data-scraper-agent",{"tool":43,"command":44},"pnpm","pnpm dlx skills add SriramGandhiS\u002FSkills --skill data-scraper-agent","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FSriramGandhiS\u002FSkills\u002Fblob\u002Fmaster\u002Feverything-claude-code\u002Fdocs\u002Fzh-CN\u002Fskills\u002Fdata-scraper-agent\u002FSKILL.md",true,[],[49,57,62,68,75,82],{"skillNo":50,"name":51,"nameZh":6,"description":52,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":53,"categories":54,"keywords":55,"publishedAt":56},"01M732609EPJWV13PVTX3GT4TM","webgpu-threejs-tsl","Comprehensive guide for developing WebGPU-enabled Three.js applications using TSL (Three.js Shading Language). Covers WebGPU renderer setup, TSL syntax and node materials, compute shaders, post-processing effects, and WGSL integration. Use this skill when working with Three.js WebGPU, TSL shaders, node materials, or GPU compute in Three.js.","https:\u002F\u002Fgithub.com\u002FSriramGandhiS\u002FSkills\u002Fblob\u002Fmaster\u002Fwebgpu-threejs-tsl\u002Fskills\u002Fwebgpu-threejs-tsl\u002FSKILL.md",[],[],"2026-11-11 15:11:14",{"skillNo":58,"name":51,"nameZh":6,"description":52,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":59,"categories":60,"keywords":61,"publishedAt":56},"01M7325XM47AGWB6K6C4VXKE6B","https:\u002F\u002Fgithub.com\u002FSriramGandhiS\u002FSkills\u002Fblob\u002Fmaster\u002Fwebgpu-threejs-tsl\u002FSKILL.md",[],[],{"skillNo":63,"name":64,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":65,"categories":66,"keywords":67,"publishedAt":56},"01M7325WP6DES69JKC840VMFEP","verification-loop","https:\u002F\u002Fgithub.com\u002FSriramGandhiS\u002FSkills\u002Fblob\u002Fmaster\u002Fweb-assets\u002Fdocs\u002Fzh-TW\u002Fskills\u002Fverification-loop\u002FSKILL.md",[],[],{"skillNo":69,"name":70,"nameZh":6,"description":71,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":72,"categories":73,"keywords":74,"publishedAt":56},"01M7325TFQF1DZTQ6J3WGQQ1VR","tdd-workflow","Use this skill when writing new features, fixing bugs, or refactoring code. Enforces test-driven development with 80%+ coverage including unit, integration, and E2E tests.","https:\u002F\u002Fgithub.com\u002FSriramGandhiS\u002FSkills\u002Fblob\u002Fmaster\u002Fweb-assets\u002Fdocs\u002Fzh-TW\u002Fskills\u002Ftdd-workflow\u002FSKILL.md",[],[],{"skillNo":76,"name":77,"nameZh":6,"description":78,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":79,"categories":80,"keywords":81,"publishedAt":56},"01M7325RGV9AY9PK92D2V8KKPW","strategic-compact","Suggests manual context compaction at logical intervals to preserve context through task phases rather than arbitrary auto-compaction.","https:\u002F\u002Fgithub.com\u002FSriramGandhiS\u002FSkills\u002Fblob\u002Fmaster\u002Fweb-assets\u002Fdocs\u002Fzh-TW\u002Fskills\u002Fstrategic-compact\u002FSKILL.md",[],[],{"skillNo":83,"name":84,"nameZh":6,"description":85,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":56},"01M7325MZC28SE86E7NMCNEFS4","security-review","Use this skill when adding authentication, handling user input, working with secrets, creating API endpoints, or implementing payment\u002Fsensitive features. Provides comprehensive security checklist and patterns.","https:\u002F\u002Fgithub.com\u002FSriramGandhiS\u002FSkills\u002Fblob\u002Fmaster\u002Fweb-assets\u002Fdocs\u002Fzh-TW\u002Fskills\u002Fsecurity-review\u002FSKILL.md",[],[]]