[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7BFACY4VRB7A7MW7FNKM4D4":3,"skill-same:01M7BFACY4VRB7A7MW7FNKM4D4":48},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":28,"repositoryHtmlUrl":29,"repositoryHomepageUrl":30,"ownerHtmlUrl":31,"ownerBlogUrl":6,"ownerDescription":32,"updatedAt":33,"install":34,"resources":47},"01M7BFACY4VRB7A7MW7FNKM4D4","data-scraper-agent",null,"构建一个全自动化的AI驱动数据收集代理，适用于任何公共来源——招聘网站、价格信息、新闻、GitHub、体育赛事等任何内容。按计划进行抓取，使用免费LLM（Gemini Flash）丰富数据，将结果存储在Notion\u002FSheets\u002FSupabase中，并从用户反馈中学习。完全免费在GitHub Actions上运行。适用于用户希望自动监控、收集或跟踪任何公共数据的场景。","YYC-Cube\u002FYYC3-Knowledge-Base","YYC-Cube","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F235006543?v=4","01M79J2PH4SWD2X0825X6PN748","yyc3","01M79J2PHVK3F6ZBYT8K2SCZCW",1,"MIT","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-AI-应用工具\u002FYYC3-AI-应用工具-Everything-claude-code\u002Fdocs\u002Fzh-CN\u002Fskills\u002Fdata-scraper-agent\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: data-scraper-agent\ndescription: 构建一个全自动化的AI驱动数据收集代理，适用于任何公共来源——招聘网站、价格信息、新闻、GitHub、体育赛事等任何内容。按计划进行抓取，使用免费LLM（Gemini Flash）丰富数据，将结果存储在Notion\u002FSheets\u002FSupabase中，并从用户反馈中学习。完全免费在GitHub Actions上运行。适用于用户希望自动监控、收集或跟踪任何公共数据的场景。\norigin: community\n---\n\n# 数据抓取代理\n\n构建一个生产就绪、AI驱动的数据收集代理，适用于任何公共数据源。\n按计划运行，使用免费LLM丰富结果，存储到数据库，并随时间推移不断改进。\n\n**技术栈：Python · Gemini Flash (免费) · GitHub Actions (免费) · Notion \u002F Sheets \u002F Supabase**\n\n## 何时激活\n\n* 用户想要抓取或监控任何公共网站或API\n* 用户说\"构建一个检查...的机器人\"、\"为我监控X\"、\"从...收集数据\"\n* 用户想要跟踪工作、价格、新闻、仓库、体育比分、事件、列表\n* 用户询问如何自动化数据收集而无需支付托管费用\n* 用户想要一个能根据他们的决策随时间推移变得更智能的代理\n\n## 核心概念\n\n### 三层架构\n\n每个数据抓取代理都有三层：\n\n```\nCOLLECT → ENRICH → STORE\n  │           │        │\nScraper    AI (LLM)  Database\nruns on    scores\u002F   Notion \u002F\nschedule   summarises Sheets \u002F\n           & classifies Supabase\n```\n\n### 免费技术栈\n\n| 层级 | 工具 | 原因 |\n|---|---|---|\n| **抓取** | `requests` + `BeautifulSoup` | 无成本，覆盖80%的公共网站 |\n| **JS渲染的网站** | `playwright` (免费) | 当HTML抓取失败时使用 |\n| **AI丰富** | 通过REST API的Gemini Flash | 500次请求\u002F天，100万令牌\u002F天 — 免费 |\n| **存储** | Notion API | 免费层级，用于审查的优秀UI |\n| **调度** | GitHub Actions cron | 对公共仓库免费 |\n| **学习** | 仓库中的JSON反馈文件 | 零基础设施，在git中持久化 |\n\n### AI模型后备链\n\n构建代理以在配额耗尽时自动在Gemini模型间回退：\n\n```\ngemini-2.0-flash-lite (30 RPM) →\ngemini-2.0-flash (15 RPM) →\ngemini-2.5-flash (10 RPM) →\ngemini-flash-lite-latest (fallback)\n```\n\n### 批量API调用以提高效率\n\n切勿为每个项目单独调用LLM。始终批量处理：\n\n```python\n# BAD: 33 API calls for 33 items\nfor item in items:\n    result = call_ai(item)  # 33 calls → hits rate limit\n\n# GOOD: 7 API calls for 33 items (batch size 5)\nfor batch in chunks(items, size=5):\n    results = call_ai(batch)  # 7 calls → stays within free tier\n```\n\n***\n\n## 工作流程\n\n### 步骤 1: 理解目标\n\n询问用户：\n\n1. **收集什么：** \"数据源是什么？URL \u002F API \u002F RSS \u002F 公共端点？\"\n2. **提取什么：** \"哪些字段重要？标题、价格、URL、日期、分数？\"\n3. **如何存储：** \"结果应该存储在哪里？Notion、Google Sheets、Supabase，还是本地文件？\"\n4. **如何丰富：** \"您希望AI对每个项目进行评分、总结、分类或匹配吗？\"\n5. **频率：** \"应该多久运行一次？每小时、每天、每周？\"\n\n常见的提示示例：\n\n* 招聘网站 → 根据简历评分相关性\n* 产品价格 → 降价时发出警报\n* GitHub仓库 → 总结新版本\n* 新闻源 → 按主题+情感分类\n* 体育结果 → 提取统计数据到跟踪器\n* 活动日历 → 按兴趣筛选\n\n***\n\n### 步骤 2: 设计代理架构\n\n为用户生成以下目录结构：\n\n```\nmy-agent\u002F\n├── config.yaml              # 用户自定义此文件（关键词、过滤器、偏好设置）\n├── profile\u002F\n│   └── context.md           # AI 使用的用户上下文（简历、兴趣、标准）\n├── scraper\u002F\n│   ├── __init__.py\n│   ├── main.py              # 协调器：抓取 → 丰富 → 存储\n│   ├── filters.py           # 基于规则的预过滤器（快速，在 AI 处理之前）\n│   └── sources\u002F\n│       ├── __init__.py\n│       └── source_name.py   # 每个数据源一个文件\n├── ai\u002F\n│   ├── __init__.py\n│   ├── client.py            # Gemini REST 客户端，带模型回退\n│   ├── pipeline.py          # 批量 AI 分析\n│   ├── jd_fetcher.py        # 从 URL 获取完整内容（可选）\n│   └── memory.py            # 从用户反馈中学习\n├── storage\u002F\n│   ├── __init__.py\n│   └── notion_sync.py       # 或 sheets_sync.py \u002F supabase_sync.py\n├── data\u002F\n│   └── feedback.json        # 用户决策历史（自动更新）\n├── .env.example\n├── setup.py                 # 一次性数据库\u002F模式创建\n├── enrich_existing.py       # 对旧行进行 AI 分数回填\n├── requirements.txt\n└── .github\u002F\n    └── workflows\u002F\n        └── scraper.yml      # GitHub Actions 计划任务\n```\n\n***\n\n### 步骤 3: 构建抓取器源\n\n适用于任何数据源的模板：\n\n```python\n# scraper\u002Fsources\u002Fmy_source.py\n\"\"\"\n[Source Name] — scrapes [what] from [where].\nMethod: [REST API \u002F HTML scraping \u002F RSS feed]\n\"\"\"\nimport requests\nfrom bs4 import BeautifulSoup\nfrom datetime import datetime, timezone\nfrom scraper.filters import is_relevant\n\nHEADERS = {\n    \"User-Agent\": \"Mozilla\u002F5.0 (compatible; research-bot\u002F1.0)\",\n}\n\n\ndef fetch() -> list[dict]:\n    \"\"\"\n    Returns a list of items with consistent schema.\n    Each item must have at minimum: name, url, date_found.\n    \"\"\"\n    results = []\n\n    # ---- REST API source ----\n    resp = requests.get(\"https:\u002F\u002Fapi.example.com\u002Fitems\", headers=HEADERS, timeout=15)\n    if resp.status_code == 200:\n        for item in resp.json().get(\"results\", []):\n            if not is_relevant(item.get(\"title\", \"\")):\n                continue\n            results.append(_normalise(item))\n\n    return results\n\n\ndef _normalise(raw: dict) -> dict:\n    \"\"\"Convert raw API\u002FHTML data to the standard schema.\"\"\"\n    return {\n        \"name\": raw.get(\"title\", \"\"),\n        \"url\": raw.get(\"link\", \"\"),\n        \"source\": \"MySource\",\n        \"date_found\": datetime.now(timezone.utc).date().isoformat(),\n        # add domain-specific fields here\n    }\n```\n\n**HTML抓取模式：**\n\n```python\nsoup = BeautifulSoup(resp.text, \"lxml\")\nfor card in soup.select(\"[class*='listing']\"):\n    title = card.select_one(\"h2, h3\").get_text(strip=True)\n    link = card.select_one(\"a\")[\"href\"]\n    if not link.startswith(\"http\"):\n        link = f\"https:\u002F\u002Fexample.com{link}\"\n```\n\n**RSS源模式：**\n\n```python\nimport xml.etree.ElementTree as ET\nroot = ET.fromstring(resp.text)\nfor item in root.findall(\".\u002F\u002Fitem\"):\n    title = item.findtext(\"title\", \"\")\n    link = item.findtext(\"link\", \"\")\n```\n\n***\n\n### 步骤 4: 构建Gemini AI客户端\n\n````python\n# ai\u002Fclient.py\nimport os, json, time, requests\n\n_last_call = 0.0\n\nMODEL_FALLBACK = [\n    \"gemini-2.0-flash-lite\",\n    \"gemini-2.0-flash\",\n    \"gemini-2.5-flash\",\n    \"gemini-flash-lite-latest\",\n]\n\n\ndef generate(prompt: str, model: str = \"\", rate_limit: float = 7.0) -> dict:\n    \"\"\"Call Gemini with auto-fallback on 429. Returns parsed JSON or {}.\"\"\"\n    global _last_call\n\n    api_key = os.environ.get(\"GEMINI_API_KEY\", \"\")\n    if not api_key:\n        return {}\n\n    elapsed = time.time() - _last_call\n    if elapsed \u003C rate_limit:\n        time.sleep(rate_limit - elapsed)\n\n    models = [model] + [m for m in MODEL_FALLBACK if m != model] if model else MODEL_FALLBACK\n    _last_call = time.time()\n\n    for m in models:\n        url = f\"https:\u002F\u002Fgenerativelanguage.googleapis.com\u002Fv1beta\u002Fmodels\u002F{m}:generateContent?key={api_key}\"\n        payload = {\n            \"contents\": [{\"parts\": [{\"text\": prompt}]}],\n            \"generationConfig\": {\n                \"responseMimeType\": \"application\u002Fjson\",\n                \"temperature\": 0.3,\n                \"maxOutputTokens\": 2048,\n            },\n        }\n        try:\n            resp = requests.post(url, json=payload, timeout=30)\n            if resp.status_code == 200:\n                return _parse(resp)\n            if resp.status_code in (429, 404):\n                time.sleep(1)\n                continue\n            return {}\n        except requests.RequestException:\n            return {}\n\n    return {}\n\n\ndef _parse(resp) -> dict:\n    try:\n        text = (\n            resp.json()\n            .get(\"candidates\", [{}])[0]\n            .get(\"content\", {})\n            .get(\"parts\", [{}])[0]\n            .get(\"text\", \"\")\n            .strip()\n        )\n        if text.startswith(\"```\"):\n            text = text.split(\"\\n\", 1)[-1].rsplit(\"```\", 1)[0]\n        return json.loads(text)\n    except (json.JSONDecodeError, KeyError):\n        return {}\n````\n\n***\n\n### 步骤 5: 构建AI管道（批量）\n\n```python\n# ai\u002Fpipeline.py\nimport json\nimport yaml\nfrom pathlib import Path\nfrom ai.client import generate\n\ndef analyse_batch(items: list[dict], context: str = \"\", preference_prompt: str = \"\") -> list[dict]:\n    \"\"\"Analyse items in batches. Returns items enriched with AI fields.\"\"\"\n    config = yaml.safe_load((Path(__file__).parent.parent \u002F \"config.yaml\").read_text())\n    model = config.get(\"ai\", {}).get(\"model\", \"gemini-2.5-flash\")\n    rate_limit = config.get(\"ai\", {}).get(\"rate_limit_seconds\", 7.0)\n    min_score = config.get(\"ai\", {}).get(\"min_score\", 0)\n    batch_size = config.get(\"ai\", {}).get(\"batch_size\", 5)\n\n    batches = [items[i:i + batch_size] for i in range(0, len(items), batch_size)]\n    print(f\"  [AI] {len(items)} items → {len(batches)} API calls\")\n\n    enriched = []\n    for i, batch in enumerate(batches):\n        print(f\"  [AI] Batch {i + 1}\u002F{len(batches)}...\")\n        prompt = _build_prompt(batch, context, preference_prompt, config)\n        result = generate(prompt, model=model, rate_limit=rate_limit)\n\n        analyses = result.get(\"analyses\", [])\n        for j, item in enumerate(batch):\n            ai = analyses[j] if j \u003C len(analyses) else {}\n            if ai:\n                score = max(0, min(100, int(ai.get(\"score\", 0))))\n                if min_score and score \u003C min_score:\n                    continue\n                enriched.append({**item, \"ai_score\": score, \"ai_summary\": ai.get(\"summary\", \"\"), \"ai_notes\": ai.get(\"notes\", \"\")})\n            else:\n                enriched.append(item)\n\n    return enriched\n\n\ndef _build_prompt(batch, context, preference_prompt, config):\n    priorities = config.get(\"priorities\", [])\n    items_text = \"\\n\\n\".join(\n        f\"Item {i+1}: {json.dumps({k: v for k, v in item.items() if not k.startswith('_')})}\"\n        for i, item in enumerate(batch)\n    )\n\n    return f\"\"\"Analyse these {len(batch)} items and return a JSON object.\n\n# Items\n{items_text}\n\n# User Context\n{context[:800] if context else \"Not provided\"}\n\n# User Priorities\n{chr(10).join(f\"- {p}\" for p in priorities)}\n\n{preference_prompt}\n\n# Instructions\nReturn: {{\"analyses\": [{{\"score\": \u003C0-100>, \"summary\": \"\u003C2 sentences>\", \"notes\": \"\u003Cwhy this matches or doesn't>\"}} for each item in order]}}\nBe concise. Score 90+=excellent match, 70-89=good, 50-69=ok, \u003C50=weak.\"\"\"\n```\n\n***\n\n### 步骤 6: 构建反馈学习系统\n\n```python\n# ai\u002Fmemory.py\n\"\"\"Learn from user decisions to improve future scoring.\"\"\"\nimport json\nfrom pathlib import Path\n\nFEEDBACK_PATH = Path(__file__).parent.parent \u002F \"data\" \u002F \"feedback.json\"\n\n\ndef load_feedback() -> dict:\n    if FEEDBACK_PATH.exists():\n        try:\n            return json.loads(FEEDBACK_PATH.read_text())\n        except (json.JSONDecodeError, OSError):\n            pass\n    return {\"positive\": [], \"negative\": []}\n\n\ndef save_feedback(fb: dict):\n    FEEDBACK_PATH.parent.mkdir(parents=True, exist_ok=True)\n    FEEDBACK_PATH.write_text(json.dumps(fb, indent=2))\n\n\ndef build_preference_prompt(feedback: dict, max_examples: int = 15) -> str:\n    \"\"\"Convert feedback history into a prompt bias section.\"\"\"\n    lines = []\n    if feedback.get(\"positive\"):\n        lines.append(\"# Items the user LIKED (positive signal):\")\n        for e in feedback[\"positive\"][-max_examples:]:\n            lines.append(f\"- {e}\")\n    if feedback.get(\"negative\"):\n        lines.append(\"\\n# Items the user SKIPPED\u002FREJECTED (negative signal):\")\n        for e in feedback[\"negative\"][-max_examples:]:\n            lines.append(f\"- {e}\")\n    if lines:\n        lines.append(\"\\nUse these patterns to bias scoring on new items.\")\n    return \"\\n\".join(lines)\n```\n\n**与存储层集成：** 每次运行后，从数据库中查询具有正面\u002F负面状态的项，并使用提取的模式调用 `save_feedback()`。\n\n***\n\n### 步骤 7: 构建存储（Notion示例）\n\n```python\n# storage\u002Fnotion_sync.py\nimport os\nfrom notion_client import Client\nfrom notion_client.errors import APIResponseError\n\n_client = None\n\ndef get_client():\n    global _client\n    if _client is None:\n        _client = Client(auth=os.environ[\"NOTION_TOKEN\"])\n    return _client\n\ndef get_existing_urls(db_id: str) -> set[str]:\n    \"\"\"Fetch all URLs already stored — used for deduplication.\"\"\"\n    client, seen, cursor = get_client(), set(), None\n    while True:\n        resp = client.databases.query(database_id=db_id, page_size=100, **{\"start_cursor\": cursor} if cursor else {})\n        for page in resp[\"results\"]:\n            url = page[\"properties\"].get(\"URL\", {}).get(\"url\", \"\")\n            if url: seen.add(url)\n        if not resp[\"has_more\"]: break\n        cursor = resp[\"next_cursor\"]\n    return seen\n\ndef push_item(db_id: str, item: dict) -> bool:\n    \"\"\"Push one item to Notion. Returns True on success.\"\"\"\n    props = {\n        \"Name\": {\"title\": [{\"text\": {\"content\": item.get(\"name\", \"\")[:100]}}]},\n        \"URL\": {\"url\": item.get(\"url\")},\n        \"Source\": {\"select\": {\"name\": item.get(\"source\", \"Unknown\")}},\n        \"Date Found\": {\"date\": {\"start\": item.get(\"date_found\")}},\n        \"Status\": {\"select\": {\"name\": \"New\"}},\n    }\n    # AI fields\n    if item.get(\"ai_score\") is not None:\n        props[\"AI Score\"] = {\"number\": item[\"ai_score\"]}\n    if item.get(\"ai_summary\"):\n        props[\"Summary\"] = {\"rich_text\": [{\"text\": {\"content\": item[\"ai_summary\"][:2000]}}]}\n    if item.get(\"ai_notes\"):\n        props[\"Notes\"] = {\"rich_text\": [{\"text\": {\"content\": item[\"ai_notes\"][:2000]}}]}\n\n    try:\n        get_client().pages.create(parent={\"database_id\": db_id}, properties=props)\n        return True\n    except APIResponseError as e:\n        print(f\"[notion] Push failed: {e}\")\n        return False\n\ndef sync(db_id: str, items: list[dict]) -> tuple[int, int]:\n    existing = get_existing_urls(db_id)\n    added = skipped = 0\n    for item in items:\n        if item.get(\"url\") in existing:\n            skipped += 1; continue\n        if push_item(db_id, item):\n            added += 1; existing.add(item[\"url\"])\n        else:\n            skipped += 1\n    return added, skipped\n```\n\n***\n\n### 步骤 8: 在 main.py 中编排\n\n```python\n# scraper\u002Fmain.py\nimport os, sys, yaml\nfrom pathlib import Path\nfrom dotenv import load_dotenv\n\nload_dotenv()\n\nfrom scraper.sources import my_source          # add your sources\n\n# NOTE: This example uses Notion. If storage.provider is \"sheets\" or \"supabase\",\n# replace this import with storage.sheets_sync or storage.supabase_sync and update\n# the env var and sync() call accordingly.\nfrom storage.notion_sync import sync\n\nSOURCES = [\n    (\"My Source\", my_source.fetch),\n]\n\ndef ai_enabled():\n    return bool(os.environ.get(\"GEMINI_API_KEY\"))\n\ndef main():\n    config = yaml.safe_load((Path(__file__).parent.parent \u002F \"config.yaml\").read_text())\n    provider = config.get(\"storage\", {}).get(\"provider\", \"notion\")\n\n    # Resolve the storage target identifier from env based on provider\n    if provider == \"notion\":\n        db_id = os.environ.get(\"NOTION_DATABASE_ID\")\n        if not db_id:\n            print(\"ERROR: NOTION_DATABASE_ID not set\"); sys.exit(1)\n    else:\n        # Extend here for sheets (SHEET_ID) or supabase (SUPABASE_TABLE) etc.\n        print(f\"ERROR: provider '{provider}' not yet wired in main.py\"); sys.exit(1)\n\n    config = yaml.safe_load((Path(__file__).parent.parent \u002F \"config.yaml\").read_text())\n    all_items = []\n\n    for name, fetch_fn in SOURCES:\n        try:\n            items = fetch_fn()\n            print(f\"[{name}] {len(items)} items\")\n            all_items.extend(items)\n        except Exception as e:\n            print(f\"[{name}] FAILED: {e}\")\n\n    # Deduplicate by URL\n    seen, deduped = set(), []\n    for item in all_items:\n        if (url := item.get(\"url\", \"\")) and url not in seen:\n            seen.add(url); deduped.append(item)\n\n    print(f\"Unique items: {len(deduped)}\")\n\n    if ai_enabled() and deduped:\n        from ai.memory import load_feedback, build_preference_prompt\n        from ai.pipeline import analyse_batch\n\n        # load_feedback() reads data\u002Ffeedback.json written by your feedback sync script.\n        # To keep it current, implement a separate feedback_sync.py that queries your\n        # storage provider for items with positive\u002Fnegative statuses and calls save_feedback().\n        feedback = load_feedback()\n        preference = build_preference_prompt(feedback)\n        context_path = Path(__file__).parent.parent \u002F \"profile\" \u002F \"context.md\"\n        context = context_path.read_text() if context_path.exists() else \"\"\n        deduped = analyse_batch(deduped, context=context, preference_prompt=preference)\n    else:\n        print(\"[AI] Skipped — GEMINI_API_KEY not set\")\n\n    added, skipped = sync(db_id, deduped)\n    print(f\"Done — {added} new, {skipped} existing\")\n\nif __name__ == \"__main__\":\n    main()\n```\n\n***\n\n### 步骤 9: GitHub Actions工作流\n\n```yaml\n# .github\u002Fworkflows\u002Fscraper.yml\nname: Data Scraper Agent\n\non:\n  schedule:\n    - cron: \"0 *\u002F3 * * *\"  # every 3 hours — adjust to your needs\n  workflow_dispatch:        # allow manual trigger\n\npermissions:\n  contents: write   # required for the feedback-history commit step\n\njobs:\n  scrape:\n    runs-on: ubuntu-latest\n    timeout-minutes: 20\n\n    steps:\n      - uses: actions\u002Fcheckout@v4\n\n      - uses: actions\u002Fsetup-python@v5\n        with:\n          python-version: \"3.11\"\n          cache: \"pip\"\n\n      - run: pip install -r requirements.txt\n\n      # Uncomment if Playwright is enabled in requirements.txt\n      # - name: Install Playwright browsers\n      #   run: python -m playwright install chromium --with-deps\n\n      - name: Run agent\n        env:\n          NOTION_TOKEN: ${{ secrets.NOTION_TOKEN }}\n          NOTION_DATABASE_ID: ${{ secrets.NOTION_DATABASE_ID }}\n          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}\n        run: python -m scraper.main\n\n      - name: Commit feedback history\n        run: |\n          git config user.name \"github-actions[bot]\"\n          git config user.email \"github-actions[bot]@users.noreply.github.com\"\n          git add data\u002Ffeedback.json || true\n          git diff --cached --quiet || git commit -m \"chore: update feedback history\"\n          git push\n```\n\n***\n\n### 步骤 10: config.yaml 模板\n\n```yaml\n# Customise this file — no code changes needed\n\n# What to collect (pre-filter before AI)\nfilters:\n  required_keywords: []      # item must contain at least one\n  blocked_keywords: []       # item must not contain any\n\n# Your priorities — AI uses these for scoring\npriorities:\n  - \"example priority 1\"\n  - \"example priority 2\"\n\n# Storage\nstorage:\n  provider: \"notion\"         # notion | sheets | supabase | sqlite\n\n# Feedback learning\nfeedback:\n  positive_statuses: [\"Saved\", \"Applied\", \"Interested\"]\n  negative_statuses: [\"Skip\", \"Rejected\", \"Not relevant\"]\n\n# AI settings\nai:\n  enabled: true\n  model: \"gemini-2.5-flash\"\n  min_score: 0               # filter out items below this score\n  rate_limit_seconds: 7      # seconds between API calls\n  batch_size: 5              # items per API call\n```\n\n***\n\n## 常见抓取模式\n\n### 模式 1: REST API（最简单）\n\n```python\nresp = requests.get(url, params={\"q\": query}, headers=HEADERS, timeout=15)\nitems = resp.json().get(\"results\", [])\n```\n\n### 模式 2: HTML抓取\n\n```python\nsoup = BeautifulSoup(resp.text, \"lxml\")\nfor card in soup.select(\".listing-card\"):\n    title = card.select_one(\"h2\").get_text(strip=True)\n    href = card.select_one(\"a\")[\"href\"]\n```\n\n### 模式 3: RSS源\n\n```python\nimport xml.etree.ElementTree as ET\nroot = ET.fromstring(resp.text)\nfor item in root.findall(\".\u002F\u002Fitem\"):\n    title = item.findtext(\"title\", \"\")\n    link = item.findtext(\"link\", \"\")\n    pub_date = item.findtext(\"pubDate\", \"\")\n```\n\n### 模式 4: 分页API\n\n```python\npage = 1\nwhile True:\n    resp = requests.get(url, params={\"page\": page, \"limit\": 50}, timeout=15)\n    data = resp.json()\n    items = data.get(\"results\", [])\n    if not items:\n        break\n    for item in items:\n        results.append(_normalise(item))\n    if not data.get(\"has_more\"):\n        break\n    page += 1\n```\n\n### 模式 5: JS渲染页面（Playwright）\n\n```python\nfrom playwright.sync_api import sync_playwright\n\nwith sync_playwright() as p:\n    browser = p.chromium.launch()\n    page = browser.new_page()\n    page.goto(url)\n    page.wait_for_selector(\".listing\")\n    html = page.content()\n    browser.close()\n\nsoup = BeautifulSoup(html, \"lxml\")\n```\n\n***\n\n## 需要避免的反模式\n\n| 反模式 | 问题 | 修复方法 |\n|---|---|---|\n| 每个项目调用一次LLM | 立即达到速率限制 | 每次调用批量处理5个项目 |\n| 代码中硬编码关键字 | 不可重用 | 将所有配置移动到 `config.yaml` |\n| 没有速率限制的抓取 | IP被禁止 | 在请求之间添加 `time.sleep(1)` |\n| 在代码中存储密钥 | 安全风险 | 始终使用 `.env` + GitHub Secrets |\n| 没有去重 | 重复行堆积 | 在推送前始终检查URL |\n| 忽略 `robots.txt` | 法律\u002F道德风险 | 遵守爬虫规则；尽可能使用公共API |\n| 使用 `requests` 处理JS渲染的网站 | 空响应 | 使用Playwright或查找底层API |\n| `maxOutputTokens` 太低 | JSON截断，解析错误 | 对批量响应使用2048+ |\n\n***\n\n## 免费层级限制参考\n\n| 服务 | 免费限制 | 典型用法 |\n|---|---|---|\n| Gemini Flash Lite | 30 RPM, 1500 RPD | 以3小时间隔约56次请求\u002F天 |\n| Gemini 2.0 Flash | 15 RPM, 1500 RPD | 良好的后备选项 |\n| Gemini 2.5 Flash | 10 RPM, 500 RPD | 谨慎使用 |\n| GitHub Actions | 无限（公共仓库） | 约20分钟\u002F天 |\n| Notion API | 无限 | 约200次写入\u002F天 |\n| Supabase | 500MB DB, 2GB传输 | 适用于大多数代理 |\n| Google Sheets API | 300次请求\u002F分钟 | 适用于小型代理 |\n\n***\n\n## 需求模板\n\n```\nrequests==2.31.0\nbeautifulsoup4==4.12.3\nlxml==5.1.0\npython-dotenv==1.0.1\npyyaml==6.0.2\nnotion-client==2.2.1   # 如需使用 Notion\n# playwright==1.40.0   # 针对 JS 渲染的站点，请取消注释\n```\n\n***\n\n## 质量检查清单\n\n在将代理标记为完成之前：\n\n* \\[ ] `config.yaml` 控制所有面向用户的设置 — 没有硬编码的值\n* \\[ ] `profile\u002Fcontext.md` 保存用于AI匹配的用户特定上下文\n* \\[ ] 在每次存储推送前通过URL进行去重\n* \\[ ] Gemini客户端具有模型后备链（4个模型）\n* \\[ ] 批量大小 ≤ 每个API调用5个项目\n* \\[ ] `maxOutputTokens` ≥ 2048\n* \\[ ] `.env` 在 `.gitignore` 中\n* \\[ ] 提供了用于入门的 `.env.example`\n* \\[ ] `setup.py` 在首次运行时创建数据库模式\n* \\[ ] `enrich_existing.py` 回填旧行的AI分数\n* \\[ ] GitHub Actions工作流在每次运行后提交 `feedback.json`\n* \\[ ] README涵盖：在\u003C5分钟内设置，所需的密钥，自定义\n\n***\n\n## 真实世界示例\n\n```\n\"为我构建一个监控 Hacker News 上 AI 初创公司融资新闻的智能体\"\n\"从 3 家电商网站抓取产品价格并在降价时发出提醒\"\n\"追踪标记有 'llm' 或 'agents' 的新 GitHub 仓库——并为每个仓库生成摘要\"\n\"将 LinkedIn 和 Cutshort 上的首席运营官职位列表收集到 Notion 中\"\n\"监控一个提到我公司的 subreddit 帖子——并进行情感分类\"\n\"每日从 arXiv 抓取我关注主题的新学术论文\"\n\"追踪体育赛事结果并在 Google Sheets 中维护动态更新的表格\"\n\"构建一个房地产房源监控器——在新房源价格低于 1 千万卢比时发出提醒\"\n```\n\n***\n\n## 参考实现\n\n一个使用此确切架构构建的完整工作代理将抓取4+个数据源，\n批量处理Gemini调用，从存储在Notion中的\"已应用\"\u002F\"已拒绝\"决策中学习，并且\n在GitHub Actions上100%免费运行。按照上述步骤1-9构建您自己的代理。\n","# 数据抓取代理\n\n构建一个生产就绪、AI驱动的数据收集代理，适用于任何公共数据源。\n按计划运行，使用免费LLM丰富结果，存储到数据库，并随时间推移不断改进。\n\n**技术栈：Python · Gemini Flash (免费) · GitHub Actions (免费) · Notion \u002F Sheets \u002F Supabase**\n\n## 何时激活\n\n* 用户想要抓取或监控任何公共网站或API\n* 用户说\"构建一个检查...的机器人\"、\"为我监控X\"、\"从...收集数据\"\n* 用户想要跟踪工作、价格、新闻、仓库、体育比分、事件、列表\n* 用户询问如何自动化数据收集而无需支付托管费用\n* 用户想要一个能根据他们的决策随时间推移变得更智能的代理\n\n## 核心概念\n\n### 三层架构\n\n每个数据抓取代理都有三层：\n\n```\nCOLLECT → ENRICH → STORE\n  │           │        │\nScraper    AI (LLM)  Database\nruns on    scores\u002F   Notion \u002F\nschedule   summarises Sheets \u002F\n           & classifies Supabase\n```\n\n### 免费技术栈\n\n| 层级 | 工具 | 原因 |\n|---|---|---|\n| **抓取** | `requests` + `BeautifulSoup` | 无成本，覆盖80%的公共网站 |\n| **JS渲染的网站** | `playwright` (免费) | 当HTML抓取失败时使用 |\n| **AI丰富** | 通过REST API的Gemini Flash | 500次请求\u002F天，100万令牌\u002F天 — 免费 |\n| **存储** | Notion API | 免费层级，用于审查的优秀UI |\n| **调度** | GitHub Actions cron | 对公共仓库免费 |\n| **学习** | 仓库中的JSON反馈文件 | 零基础设施，在git中持久化 |\n\n### AI模型后备链\n\n构建代理以在配额耗尽时自动在Gemini模型间回退：\n\n```\ngemini-2.0-flash-lite (30 RPM) →\ngemini-2.0-flash (15 RPM) →\ngemini-2.5-flash (10 RPM) →\ngemini-flash-lite-latest (fallback)\n```\n\n### 批量API调用以提高效率\n\n切勿为每个项目单独调用LLM。始终批量处理：\n\n```python\n# BAD: 33 API calls for 33 items\nfor item in items:\n    result = call_ai(item)  # 33 calls → hits rate limit\n\n# GOOD: 7 API calls for 33 items (batch size 5)\nfor batch in chunks(items, size=5):\n    results = call_ai(batch)  # 7 calls → stays within free tier\n```\n\n***\n\n## 工作流程\n\n### 步骤 1: 理解目标\n\n询问用户：\n\n1. **收集什么：** \"数据源是什么？URL \u002F API \u002F RSS \u002F 公共端点？\"\n2. **提取什么：** \"哪些字段重要？标题、价格、URL、日期、分数？\"\n3. **如何存储：** \"结果应该存储在哪里？Notion、Google Sheets、Supabase，还是本地文件？\"\n4. **如何丰富：** \"您希望AI对每个项目进行评分、总结、分类或匹配吗？\"\n5. **频率：** \"应该多久运行一次？每小时、每天、每周？\"\n\n常见的提示示例：\n\n* 招聘网站 → 根据简历评分相关性\n* 产品价格 → 降价时发出警报\n* GitHub仓库 → 总结新版本\n* 新闻源 → 按主题+情感分类\n* 体育结果 → 提取统计数据到跟踪器\n* 活动日历 → 按兴趣筛选\n\n***\n\n### 步骤 2: 设计代理架构\n\n为用户生成以下目录结构：\n\n```\nmy-agent\u002F\n├── config.yaml              # 用户自定义此文件（关键词、过滤器、偏好设置）\n├── profile\u002F\n│   └── context.md           # AI 使用的用户上下文（简历、兴趣、标准）\n├── scraper\u002F\n│   ├── __init__.py\n│   ├── main.py              # 协调器：抓取 → 丰富 → 存储\n│   ├── filters.py           # 基于规则的预过滤器（快速，在 AI 处理之前）\n│   └── sources\u002F\n│       ├── __init__.py\n│       └── source_name.py   # 每个数据源一个文件\n├── ai\u002F\n│   ├── __init__.py\n│   ├── client.py            # Gemini REST 客户端，带模型回退\n│   ├── pipeline.py          # 批量 AI 分析\n│   ├── jd_fetcher.py        # 从 URL 获取完整内容（可选）\n│   └── memory.py            # 从用户反馈中学习\n├── storage\u002F\n│   ├── __init__.py\n│   └── notion_sync.py       # 或 sheets_sync.py \u002F supabase_sync.py\n├── data\u002F\n│   └── feedback.json        # 用户决策历史（自动更新）\n├── .env.example\n├── setup.py                 # 一次性数据库\u002F模式创建\n├── enrich_existing.py       # 对旧行进行 AI 分数回填\n├── requirements.txt\n└── .github\u002F\n    └── workflows\u002F\n        └── scraper.yml      # GitHub Actions 计划任务\n```\n\n***\n\n### 步骤 3: 构建抓取器源\n\n适用于任何数据源的模板：\n\n```python\n# scraper\u002Fsources\u002Fmy_source.py\n\"\"\"\n[Source Name] — scrapes [what] from [where].\nMethod: [REST API \u002F HTML scraping \u002F RSS feed]\n\"\"\"\nimport requests\nfrom bs4 import BeautifulSoup\nfrom datetime import datetime, timezone\nfrom scraper.filters import is_relevant\n\nHEADERS = {\n    \"User-Agent\": \"Mozilla\u002F5.0 (compatible; research-bot\u002F1.0)\",\n}\n\n\ndef fetch() -> list[dict]:\n    \"\"\"\n    Returns a list of items with consistent schema.\n    Each item must have at minimum: name, url, date_found.\n    \"\"\"\n    results = []\n\n    # ---- REST API source ----\n    resp = requests.get(\"https:\u002F\u002Fapi.example.com\u002Fitems\", headers=HEADERS, timeout=15)\n    if resp.status_code == 200:\n        for item in resp.json().get(\"results\", []):\n            if not is_relevant(item.get(\"title\", \"\")):\n                continue\n            results.append(_normalise(item))\n\n    return results\n\n\ndef _normalise(raw: dict) -> dict:\n    \"\"\"Convert raw API\u002FHTML data to the standard schema.\"\"\"\n    return {\n        \"name\": raw.get(\"title\", \"\"),\n        \"url\": raw.get(\"link\", \"\"),\n        \"source\": \"MySource\",\n        \"date_found\": datetime.now(timezone.utc).date().isoformat(),\n        # add domain-specific fields here\n    }\n```\n\n**HTML抓取模式：**\n\n```python\nsoup = BeautifulSoup(resp.text, \"lxml\")\nfor card in soup.select(\"[class*='listing']\"):\n    title = card.select_one(\"h2, h3\").get_text(strip=True)\n    link = card.select_one(\"a\")[\"href\"]\n    if not link.startswith(\"http\"):\n        link = f\"https:\u002F\u002Fexample.com{link}\"\n```\n\n**RSS源模式：**\n\n```python\nimport xml.etree.ElementTree as ET\nroot = ET.fromstring(resp.text)\nfor item in root.findall(\".\u002F\u002Fitem\"):\n    title = item.findtext(\"title\", \"\")\n    link = item.findtext(\"link\", \"\")\n```\n\n***\n\n### 步骤 4: 构建Gemini AI客户端\n\n````python\n# ai\u002Fclient.py\nimport os, json, time, requests\n\n_last_call = 0.0\n\nMODEL_FALLBACK = [\n    \"gemini-2.0-flash-lite\",\n    \"gemini-2.0-flash\",\n    \"gemini-2.5-flash\",\n    \"gemini-flash-lite-latest\",\n]\n\n\ndef generate(prompt: str, model: str = \"\", rate_limit: float = 7.0) -> dict:\n    \"\"\"Call Gemini with auto-fallback on 429. Returns parsed JSON or {}.\"\"\"\n    global _last_call\n\n    api_key = os.environ.get(\"GEMINI_API_KEY\", \"\")\n    if not api_key:\n        return {}\n\n    elapsed = time.time() - _last_call\n    if elapsed \u003C rate_limit:\n        time.sleep(rate_limit - elapsed)\n\n    models = [model] + [m for m in MODEL_FALLBACK if m != model] if model else MODEL_FALLBACK\n    _last_call = time.time()\n\n    for m in models:\n        url = f\"https:\u002F\u002Fgenerativelanguage.googleapis.com\u002Fv1beta\u002Fmodels\u002F{m}:generateContent?key={api_key}\"\n        payload = {\n            \"contents\": [{\"parts\": [{\"text\": prompt}]}],\n            \"generationConfig\": {\n                \"responseMimeType\": \"application\u002Fjson\",\n                \"temperature\": 0.3,\n                \"maxOutputTokens\": 2048,\n            },\n        }\n        try:\n            resp = requests.post(url, json=payload, timeout=30)\n            if resp.status_code == 200:\n                return _parse(resp)\n            if resp.status_code in (429, 404):\n                time.sleep(1)\n                continue\n            return {}\n        except requests.RequestException:\n            return {}\n\n    return {}\n\n\ndef _parse(resp) -> dict:\n    try:\n        text = (\n            resp.json()\n            .get(\"candidates\", [{}])[0]\n            .get(\"content\", {})\n            .get(\"parts\", [{}])[0]\n            .get(\"text\", \"\")\n            .strip()\n        )\n        if text.startswith(\"```\"):\n            text = text.split(\"\\n\", 1)[-1].rsplit(\"```\", 1)[0]\n        return json.loads(text)\n    except (json.JSONDecodeError, KeyError):\n        return {}\n````\n\n***\n\n### 步骤 5: 构建AI管道（批量）\n\n```python\n# ai\u002Fpipeline.py\nimport json\nimport yaml\nfrom pathlib import Path\nfrom ai.client import generate\n\ndef analyse_batch(items: list[dict], context: str = \"\", preference_prompt: str = \"\") -> list[dict]:\n    \"\"\"Analyse items in batches. Returns items enriched with AI fields.\"\"\"\n    config = yaml.safe_load((Path(__file__).parent.parent \u002F \"config.yaml\").read_text())\n    model = config.get(\"ai\", {}).get(\"model\", \"gemini-2.5-flash\")\n    rate_limit = config.get(\"ai\", {}).get(\"rate_limit_seconds\", 7.0)\n    min_score = config.get(\"ai\", {}).get(\"min_score\", 0)\n    batch_size = config.get(\"ai\", {}).get(\"batch_size\", 5)\n\n    batches = [items[i:i + batch_size] for i in range(0, len(items), batch_size)]\n    print(f\"  [AI] {len(items)} items → {len(batches)} API calls\")\n\n    enriched = []\n    for i, batch in enumerate(batches):\n        print(f\"  [AI] Batch {i + 1}\u002F{len(batches)}...\")\n        prompt = _build_prompt(batch, context, preference_prompt, config)\n        result = generate(prompt, model=model, rate_limit=rate_limit)\n\n        analyses = result.get(\"analyses\", [])\n        for j, item in enumerate(batch):\n            ai = analyses[j] if j \u003C len(analyses) else {}\n            if ai:\n                score = max(0, min(100, int(ai.get(\"score\", 0))))\n                if min_score and score \u003C min_score:\n                    continue\n                enriched.append({**item, \"ai_score\": score, \"ai_summary\": ai.get(\"summary\", \"\"), \"ai_notes\": ai.get(\"notes\", \"\")})\n            else:\n                enriched.append(item)\n\n    return enriched\n\n\ndef _build_prompt(batch, context, preference_prompt, config):\n    priorities = config.get(\"priorities\", [])\n    items_text = \"\\n\\n\".join(\n        f\"Item {i+1}: {json.dumps({k: v for k, v in item.items() if not k.startswith('_')})}\"\n        for i, item in enumerate(batch)\n    )\n\n    return f\"\"\"Analyse these {len(batch)} items and return a JSON object.\n\n# Items\n{items_text}\n\n# User Context\n{context[:800] if context else \"Not provided\"}\n\n# User Priorities\n{chr(10).join(f\"- {p}\" for p in priorities)}\n\n{preference_prompt}\n\n# Instructions\nReturn: {{\"analyses\": [{{\"score\": \u003C0-100>, \"summary\": \"\u003C2 sentences>\", \"notes\": \"\u003Cwhy this matches or doesn't>\"}} for each item in order]}}\nBe concise. Score 90+=excellent match, 70-89=good, 50-69=ok, \u003C50=weak.\"\"\"\n```\n\n***\n\n### 步骤 6: 构建反馈学习系统\n\n```python\n# ai\u002Fmemory.py\n\"\"\"Learn from user decisions to improve future scoring.\"\"\"\nimport json\nfrom pathlib import Path\n\nFEEDBACK_PATH = Path(__file__).parent.parent \u002F \"data\" \u002F \"feedback.json\"\n\n\ndef load_feedback() -> dict:\n    if FEEDBACK_PATH.exists():\n        try:\n            return json.loads(FEEDBACK_PATH.read_text())\n        except (json.JSONDecodeError, OSError):\n            pass\n    return {\"positive\": [], \"negative\": []}\n\n\ndef save_feedback(fb: dict):\n    FEEDBACK_PATH.parent.mkdir(parents=True, exist_ok=True)\n    FEEDBACK_PATH.write_text(json.dumps(fb, indent=2))\n\n\ndef build_preference_prompt(feedback: dict, max_examples: int = 15) -> str:\n    \"\"\"Convert feedback history into a prompt bias section.\"\"\"\n    lines = []\n    if feedback.get(\"positive\"):\n        lines.append(\"# Items the user LIKED (positive signal):\")\n        for e in feedback[\"positive\"][-max_examples:]:\n            lines.append(f\"- {e}\")\n    if feedback.get(\"negative\"):\n        lines.append(\"\\n# Items the user SKIPPED\u002FREJECTED (negative signal):\")\n        for e in feedback[\"negative\"][-max_examples:]:\n            lines.append(f\"- {e}\")\n    if lines:\n        lines.append(\"\\nUse these patterns to bias scoring on new items.\")\n    return \"\\n\".join(lines)\n```\n\n**与存储层集成：** 每次运行后，从数据库中查询具有正面\u002F负面状态的项，并使用提取的模式调用 `save_feedback()`。\n\n***\n\n### 步骤 7: 构建存储（Notion示例）\n\n```python\n# storage\u002Fnotion_sync.py\nimport os\nfrom notion_client import Client\nfrom notion_client.errors import APIResponseError\n\n_client = None\n\ndef get_client():\n    global _client\n    if _client is None:\n        _client = Client(auth=os.environ[\"NOTION_TOKEN\"])\n    return _client\n\ndef get_existing_urls(db_id: str) -> set[str]:\n    \"\"\"Fetch all URLs already stored — used for deduplication.\"\"\"\n    client, seen, cursor = get_client(), set(), None\n    while True:\n        resp = client.databases.query(database_id=db_id, page_size=100, **{\"start_cursor\": cursor} if cursor else {})\n        for page in resp[\"results\"]:\n            url = page[\"properties\"].get(\"URL\", {}).get(\"url\", \"\")\n            if url: seen.add(url)\n        if not resp[\"has_more\"]: break\n        cursor = resp[\"next_cursor\"]\n    return seen\n\ndef push_item(db_id: str, item: dict) -> bool:\n    \"\"\"Push one item to Notion. Returns True on success.\"\"\"\n    props = {\n        \"Name\": {\"title\": [{\"text\": {\"content\": item.get(\"name\", \"\")[:100]}}]},\n        \"URL\": {\"url\": item.get(\"url\")},\n        \"Source\": {\"select\": {\"name\": item.get(\"source\", \"Unknown\")}},\n        \"Date Found\": {\"date\": {\"start\": item.get(\"date_found\")}},\n        \"Status\": {\"select\": {\"name\": \"New\"}},\n    }\n    # AI fields\n    if item.get(\"ai_score\") is not None:\n        props[\"AI Score\"] = {\"number\": item[\"ai_score\"]}\n    if item.get(\"ai_summary\"):\n        props[\"Summary\"] = {\"rich_text\": [{\"text\": {\"content\": item[\"ai_summary\"][:2000]}}]}\n    if item.get(\"ai_notes\"):\n        props[\"Notes\"] = {\"rich_text\": [{\"text\": {\"content\": item[\"ai_notes\"][:2000]}}]}\n\n    try:\n        get_client().pages.create(parent={\"database_id\": db_id}, properties=props)\n        return True\n    except APIResponseError as e:\n        print(f\"[notion] Push failed: {e}\")\n        return False\n\ndef sync(db_id: str, items: list[dict]) -> tuple[int, int]:\n    existing = get_existing_urls(db_id)\n    added = skipped = 0\n    for item in items:\n        if item.get(\"url\") in existing:\n            skipped += 1; continue\n        if push_item(db_id, item):\n            added += 1; existing.add(item[\"url\"])\n        else:\n            skipped += 1\n    return added, skipped\n```\n\n***\n\n### 步骤 8: 在 main.py 中编排\n\n```python\n# scraper\u002Fmain.py\nimport os, sys, yaml\nfrom pathlib import Path\nfrom dotenv import load_dotenv\n\nload_dotenv()\n\nfrom scraper.sources import my_source          # add your sources\n\n# NOTE: This example uses Notion. If storage.provider is \"sheets\" or \"supabase\",\n# replace this import with storage.sheets_sync or storage.supabase_sync and update\n# the env var and sync() call accordingly.\nfrom storage.notion_sync import sync\n\nSOURCES = [\n    (\"My Source\", my_source.fetch),\n]\n\ndef ai_enabled():\n    return bool(os.environ.get(\"GEMINI_API_KEY\"))\n\ndef main():\n    config = yaml.safe_load((Path(__file__).parent.parent \u002F \"config.yaml\").read_text())\n    provider = config.get(\"storage\", {}).get(\"provider\", \"notion\")\n\n    # Resolve the storage target identifier from env based on provider\n    if provider == \"notion\":\n        db_id = os.environ.get(\"NOTION_DATABASE_ID\")\n        if not db_id:\n            print(\"ERROR: NOTION_DATABASE_ID not set\"); sys.exit(1)\n    else:\n        # Extend here for sheets (SHEET_ID) or supabase (SUPABASE_TABLE) etc.\n        print(f\"ERROR: provider '{provider}' not yet wired in main.py\"); sys.exit(1)\n\n    config = yaml.safe_load((Path(__file__).parent.parent \u002F \"config.yaml\").read_text())\n    all_items = []\n\n    for name, fetch_fn in SOURCES:\n        try:\n            items = fetch_fn()\n            print(f\"[{name}] {len(items)} items\")\n            all_items.extend(items)\n        except Exception as e:\n            print(f\"[{name}] FAILED: {e}\")\n\n    # Deduplicate by URL\n    seen, deduped = set(), []\n    for item in all_items:\n        if (url := item.get(\"url\", \"\")) and url not in seen:\n            seen.add(url); deduped.append(item)\n\n    print(f\"Unique items: {len(deduped)}\")\n\n    if ai_enabled() and deduped:\n        from ai.memory import load_feedback, build_preference_prompt\n        from ai.pipeline import analyse_batch\n\n        # load_feedback() reads data\u002Ffeedback.json written by your feedback sync script.\n        # To keep it current, implement a separate feedback_sync.py that queries your\n        # storage provider for items with positive\u002Fnegative statuses and calls save_feedback().\n        feedback = load_feedback()\n        preference = build_preference_prompt(feedback)\n        context_path = Path(__file__).parent.parent \u002F \"profile\" \u002F \"context.md\"\n        context = context_path.read_text() if context_path.exists() else \"\"\n        deduped = analyse_batch(deduped, context=context, preference_prompt=preference)\n    else:\n        print(\"[AI] Skipped — GEMINI_API_KEY not set\")\n\n    added, skipped = sync(db_id, deduped)\n    print(f\"Done — {added} new, {skipped} existing\")\n\nif __name__ == \"__main__\":\n    main()\n```\n\n***\n\n### 步骤 9: GitHub Actions工作流\n\n```yaml\n# .github\u002Fworkflows\u002Fscraper.yml\nname: Data Scraper Agent\n\non:\n  schedule:\n    - cron: \"0 *\u002F3 * * *\"  # every 3 hours — adjust to your needs\n  workflow_dispatch:        # allow manual trigger\n\npermissions:\n  contents: write   # required for the feedback-history commit step\n\njobs:\n  scrape:\n    runs-on: ubuntu-latest\n    timeout-minutes: 20\n\n    steps:\n      - uses: actions\u002Fcheckout@v4\n\n      - uses: actions\u002Fsetup-python@v5\n        with:\n          python-version: \"3.11\"\n          cache: \"pip\"\n\n      - run: pip install -r requirements.txt\n\n      # Uncomment if Playwright is enabled in requirements.txt\n      # - name: Install Playwright browsers\n      #   run: python -m playwright install chromium --with-deps\n\n      - name: Run agent\n        env:\n          NOTION_TOKEN: ${{ secrets.NOTION_TOKEN }}\n          NOTION_DATABASE_ID: ${{ secrets.NOTION_DATABASE_ID }}\n          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}\n        run: python -m scraper.main\n\n      - name: Commit feedback history\n        run: |\n          git config user.name \"github-actions[bot]\"\n          git config user.email \"github-actions[bot]@users.noreply.github.com\"\n          git add data\u002Ffeedback.json || true\n          git diff --cached --quiet || git commit -m \"chore: update feedback history\"\n          git push\n```\n\n***\n\n### 步骤 10: config.yaml 模板\n\n```yaml\n# Customise this file — no code changes needed\n\n# What to collect (pre-filter before AI)\nfilters:\n  required_keywords: []      # item must contain at least one\n  blocked_keywords: []       # item must not contain any\n\n# Your priorities — AI uses these for scoring\npriorities:\n  - \"example priority 1\"\n  - \"example priority 2\"\n\n# Storage\nstorage:\n  provider: \"notion\"         # notion | sheets | supabase | sqlite\n\n# Feedback learning\nfeedback:\n  positive_statuses: [\"Saved\", \"Applied\", \"Interested\"]\n  negative_statuses: [\"Skip\", \"Rejected\", \"Not relevant\"]\n\n# AI settings\nai:\n  enabled: true\n  model: \"gemini-2.5-flash\"\n  min_score: 0               # filter out items below this score\n  rate_limit_seconds: 7      # seconds between API calls\n  batch_size: 5              # items per API call\n```\n\n***\n\n## 常见抓取模式\n\n### 模式 1: REST API（最简单）\n\n```python\nresp = requests.get(url, params={\"q\": query}, headers=HEADERS, timeout=15)\nitems = resp.json().get(\"results\", [])\n```\n\n### 模式 2: HTML抓取\n\n```python\nsoup = BeautifulSoup(resp.text, \"lxml\")\nfor card in soup.select(\".listing-card\"):\n    title = card.select_one(\"h2\").get_text(strip=True)\n    href = card.select_one(\"a\")[\"href\"]\n```\n\n### 模式 3: RSS源\n\n```python\nimport xml.etree.ElementTree as ET\nroot = ET.fromstring(resp.text)\nfor item in root.findall(\".\u002F\u002Fitem\"):\n    title = item.findtext(\"title\", \"\")\n    link = item.findtext(\"link\", \"\")\n    pub_date = item.findtext(\"pubDate\", \"\")\n```\n\n### 模式 4: 分页API\n\n```python\npage = 1\nwhile True:\n    resp = requests.get(url, params={\"page\": page, \"limit\": 50}, timeout=15)\n    data = resp.json()\n    items = data.get(\"results\", [])\n    if not items:\n        break\n    for item in items:\n        results.append(_normalise(item))\n    if not data.get(\"has_more\"):\n        break\n    page += 1\n```\n\n### 模式 5: JS渲染页面（Playwright）\n\n```python\nfrom playwright.sync_api import sync_playwright\n\nwith sync_playwright() as p:\n    browser = p.chromium.launch()\n    page = browser.new_page()\n    page.goto(url)\n    page.wait_for_selector(\".listing\")\n    html = page.content()\n    browser.close()\n\nsoup = BeautifulSoup(html, \"lxml\")\n```\n\n***\n\n## 需要避免的反模式\n\n| 反模式 | 问题 | 修复方法 |\n|---|---|---|\n| 每个项目调用一次LLM | 立即达到速率限制 | 每次调用批量处理5个项目 |\n| 代码中硬编码关键字 | 不可重用 | 将所有配置移动到 `config.yaml` |\n| 没有速率限制的抓取 | IP被禁止 | 在请求之间添加 `time.sleep(1)` |\n| 在代码中存储密钥 | 安全风险 | 始终使用 `.env` + GitHub Secrets |\n| 没有去重 | 重复行堆积 | 在推送前始终检查URL |\n| 忽略 `robots.txt` | 法律\u002F道德风险 | 遵守爬虫规则；尽可能使用公共API |\n| 使用 `requests` 处理JS渲染的网站 | 空响应 | 使用Playwright或查找底层API |\n| `maxOutputTokens` 太低 | JSON截断，解析错误 | 对批量响应使用2048+ |\n\n***\n\n## 免费层级限制参考\n\n| 服务 | 免费限制 | 典型用法 |\n|---|---|---|\n| Gemini Flash Lite | 30 RPM, 1500 RPD | 以3小时间隔约56次请求\u002F天 |\n| Gemini 2.0 Flash | 15 RPM, 1500 RPD | 良好的后备选项 |\n| Gemini 2.5 Flash | 10 RPM, 500 RPD | 谨慎使用 |\n| GitHub Actions | 无限（公共仓库） | 约20分钟\u002F天 |\n| Notion API | 无限 | 约200次写入\u002F天 |\n| Supabase | 500MB DB, 2GB传输 | 适用于大多数代理 |\n| Google Sheets API | 300次请求\u002F分钟 | 适用于小型代理 |\n\n***\n\n## 需求模板\n\n```\nrequests==2.31.0\nbeautifulsoup4==4.12.3\nlxml==5.1.0\npython-dotenv==1.0.1\npyyaml==6.0.2\nnotion-client==2.2.1   # 如需使用 Notion\n# playwright==1.40.0   # 针对 JS 渲染的站点，请取消注释\n```\n\n***\n\n## 质量检查清单\n\n在将代理标记为完成之前：\n\n* \\[ ] `config.yaml` 控制所有面向用户的设置 — 没有硬编码的值\n* \\[ ] `profile\u002Fcontext.md` 保存用于AI匹配的用户特定上下文\n* \\[ ] 在每次存储推送前通过URL进行去重\n* \\[ ] Gemini客户端具有模型后备链（4个模型）\n* \\[ ] 批量大小 ≤ 每个API调用5个项目\n* \\[ ] `maxOutputTokens` ≥ 2048\n* \\[ ] `.env` 在 `.gitignore` 中\n* \\[ ] 提供了用于入门的 `.env.example`\n* \\[ ] `setup.py` 在首次运行时创建数据库模式\n* \\[ ] `enrich_existing.py` 回填旧行的AI分数\n* \\[ ] GitHub Actions工作流在每次运行后提交 `feedback.json`\n* \\[ ] README涵盖：在\u003C5分钟内设置，所需的密钥，自定义\n\n***\n\n## 真实世界示例\n\n```\n\"为我构建一个监控 Hacker News 上 AI 初创公司融资新闻的智能体\"\n\"从 3 家电商网站抓取产品价格并在降价时发出提醒\"\n\"追踪标记有 'llm' 或 'agents' 的新 GitHub 仓库——并为每个仓库生成摘要\"\n\"将 LinkedIn 和 Cutshort 上的首席运营官职位列表收集到 Notion 中\"\n\"监控一个提到我公司的 subreddit 帖子——并进行情感分类\"\n\"每日从 arXiv 抓取我关注主题的新学术论文\"\n\"追踪体育赛事结果并在 Google Sheets 中维护动态更新的表格\"\n\"构建一个房地产房源监控器——在新房源价格低于 1 千万卢比时发出提醒\"\n```\n\n***\n\n## 参考实现\n\n一个使用此确切架构构建的完整工作代理将抓取4+个数据源，\n批量处理Gemini调用，从存储在Notion中的\"已应用\"\u002F\"已拒绝\"决策中学习，并且\n在GitHub Actions上100%免费运行。按照上述步骤1-9构建您自己的代理。\n",{"name":5,"origin":24,"description":7},"community","main","87afe6ce96220accf50a345dea46784c8c9f210e","YYC3-AI-应用工具\u002FYYC3-AI-应用工具-Everything-claude-code\u002Fdocs\u002Fzh-CN\u002Fskills\u002Fdata-scraper-agent",0,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base","https:\u002F\u002Fkb.yyc3.top\u002F","https:\u002F\u002Fgithub.com\u002FYYC-Cube","万象归元于云枢 | 深栈智启新纪元","2026-11-10 10:37:32",{"source":8,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill data-scraper-agent",{"tool":40,"command":41},"bunx","bunx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill data-scraper-agent",{"tool":43,"command":44},"pnpm","pnpm dlx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill data-scraper-agent","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-AI-应用工具\u002FYYC3-AI-应用工具-Everything-claude-code\u002Fdocs\u002Fzh-CN\u002Fskills\u002Fdata-scraper-agent\u002FSKILL.md",true,[],[49,56,63,70,77,84],{"skillNo":50,"name":51,"nameZh":6,"description":52,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":53,"categories":54,"keywords":55,"publishedAt":19},"01M7CTDWHERCXPQCP0K03A86AS","vercel-sandbox","Run agent-browser + Chrome inside Vercel Sandbox microVMs for browser automation from any Vercel-deployed app. Use when the user needs browser automation in a Vercel app (Next.js, SvelteKit, Nuxt, Remix, Astro, etc.), wants to run headless Chrome without binary size limits, needs persistent browser sessions across commands, or wants ephemeral isolated browser environments. Triggers include \"Vercel Sandbox browser\", \"microVM Chrome\", \"agent-browser in sandbox\", \"browser automation on Vercel\", or any task requiring Chrome in a Vercel Sandbox.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fvercel-sandbox\u002FSKILL.md",[],[],{"skillNo":57,"name":58,"nameZh":6,"description":59,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":60,"categories":61,"keywords":62,"publishedAt":19},"01M7CTD8D2BNJY54S65M47HEJA","slack","Interact with Slack workspaces using browser automation. Use when the user needs to check unread channels, navigate Slack, send messages, extract data, find information, search conversations, or automate any Slack task. Triggers include \"check my Slack\", \"what channels have unreads\", \"send a message to\", \"search Slack for\", \"extract from Slack\", \"find who said\", or any task requiring programmatic Slack interaction.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fslack\u002FSKILL.md",[],[],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":19},"01M7CTCSSB8ZP3Q58C3YZDZK1W","electron","Automate Electron desktop apps (VS Code, Slack, Discord, Figma, Notion, Spotify, etc.) using agent-browser via Chrome DevTools Protocol. Use when the user needs to interact with an Electron app, automate a desktop app, connect to a running app, control a native app, or test an Electron application. Triggers include \"automate Slack app\", \"control VS Code\", \"interact with Discord app\", \"test this Electron app\", \"connect to desktop app\", or any task requiring automation of a native Electron application.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Felectron\u002FSKILL.md",[],[],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M7CTCBJY76688V4DCZGWFHGQ","dogfood","Systematically explore and test a web application to find bugs, UX issues, and other problems. Use when asked to \"dogfood\", \"QA\", \"exploratory test\", \"find issues\", \"bug hunt\", \"test this app\u002Fsite\u002Fplatform\", or review the quality of a web application. Produces a structured report with full reproduction evidence -- step-by-step screenshots, repro videos, and detailed repro steps for every issue -- so findings can be handed directly to the responsible teams.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fdogfood\u002FSKILL.md",[],[],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":19},"01M7CTC4H0FFGVEAQR9PA4GA1A","agent-browser","Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to \"open a website\", \"fill out a form\", \"click a button\", \"take a screenshot\", \"scrape data from a page\", \"test this web app\", \"login to a site\", \"automate browser actions\", or any task requiring programmatic web interaction.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fagent-browser\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":19},"01M7CTBKVFYBG78FDSABSDJTTQ","shadcn","Manages shadcn components and projects — adding, searching, fixing, debugging, styling, and composing UI, including chat interfaces. Provides project context, component docs, and usage examples. Applies when working with shadcn\u002Fui, component registries, presets, --preset codes, or any project with a components.json file. Also triggers for \"shadcn init\", \"create an app with --preset\", \"switch to --preset\", and for building a design system or applying a DESIGN.md or brand spec to shadcn\u002Fui.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-UI-界面组件\u002FYYC3-UI-界面组件-ui\u002Fskills\u002Fshadcn\u002FSKILL.md",[],[]]