[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MHYT4VDHX4CGPRYN2Y5WZE":3,"skill-same:01M6MHYT4VDHX4CGPRYN2Y5WZE":70},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":41,"linkOnly":42,"markdownContent":43,"markdownBody":44,"frontMatter":45,"refName":46,"commitSha":47,"skillDirectory":48,"forksCount":49,"repositoryHtmlUrl":50,"repositoryHomepageUrl":51,"ownerHtmlUrl":52,"ownerBlogUrl":53,"ownerDescription":54,"updatedAt":55,"install":56,"resources":69},"01M6MHYT4VDHX4CGPRYN2Y5WZE","crawling-a-site",null,"Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers `crawlberg crawl` with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1575,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fkreuzberg-dev\u002Fplugins\u002Fplugins\u002Fcrawlberg\u002Fskills\u002Fcrawling-a-site\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40],"site","crawling a site","crawling","rate limiting","when","depth","crawlberg","rate","links","with","output","as","concurrency","across","domain","crawl","selection","use","scoping","capture","page","markdown","2026-11-10 10:12:30",false,"---\nname: crawling-a-site\ndescription: >-\n  Use when the user wants to follow links across a domain and capture every\n  reachable page as Markdown. Covers `crawlberg crawl` with depth, page\n  caps, concurrency, rate limiting, domain scoping, robots, and output\n  selection.\n---\n\n\u003C!--\nAI-RULEZ :: GENERATED FILE — DO NOT EDIT\nContent-Hash: blake3:87e62f8235715c6d9ff820d40d26b521d5d0e0fb9888f7d04589c7429d28da14\nSource-Hash: blake3:b5689383a3914da8e3cc6ad3614356ae7f98c90cf99cb1b4b47be38456ff7a7c\nSchema-Version: v1\n-->\n\n# Crawling a site\n\nReach for `crawlberg crawl` when one URL is not enough — the user wants\nthe docs site, the blog, the marketing pages, or the whole domain.\n\n## Quick recipe\n\n```bash\ncrawlberg crawl https:\u002F\u002Fexample.com \\\n  --depth 3 \\\n  --max-pages 200 \\\n  --concurrent 8 \\\n  --rate-limit 250 \\\n  --stay-on-domain \\\n  --respect-robots-txt \\\n  --format markdown\n```\n\nDefaults you should usually override:\n\n- `--depth 2` is shallow — set it explicitly.\n- `--max-pages` is unbounded by default; cap it for any unknown site.\n- `--concurrent 10` is aggressive for small hosts; drop to 4-8 for\n  third-party sites.\n\n## Flag surface\n\n| Flag                    | Default | Purpose                                                    |\n| ----------------------- | ------- | ---------------------------------------------------------- |\n| `--depth`, `-d`         | `2`     | Maximum hop count from the seed URL.                       |\n| `--max-pages`, `-n`     | —       | Hard cap on pages fetched. Set this on any unknown site.   |\n| `--concurrent`, `-c`    | `10`    | Parallel in-flight requests.                               |\n| `--rate-limit`          | `200`   | Milliseconds between requests to the same origin.          |\n| `--stay-on-domain`      | off     | Skip links that leave the seed domain.                     |\n| `--respect-robots-txt`  | off     | Honour `robots.txt`. Pass it for any third-party host.     |\n| `--proxy`               | —       | HTTP, HTTPS, or SOCKS5 proxy URL.                          |\n| `--user-agent`          | —       | Override the request UA. Be honest.                        |\n| `--timeout`             | `30000` | Per-request timeout in ms.                                 |\n| `--browser-mode`        | `auto`  | `auto`, `always`, `never` — see the headless-fallback skill. |\n| `--browser-endpoint`    | —       | External CDP `ws:\u002F\u002F` URL.                                  |\n| `--format`              | `json`  | `json` or `markdown`.                                      |\n| `--config`              | —       | Inline JSON or `@file.json` for the full `CrawlConfig`.    |\n\nMultiple seed URLs are accepted positionally — the engine fans out with\n`batch_crawl` and aggregates results.\n\n## When to pick which flags\n\n### Docs sites you own\n\n```bash\ncrawlberg crawl https:\u002F\u002Fdocs.example.com \\\n  --depth 5 --max-pages 1000 --concurrent 16 --rate-limit 100 \\\n  --stay-on-domain --format markdown > docs.md\n```\n\nHigher concurrency and lower rate limits are fine on infrastructure you\ncontrol.\n\n### Third-party sites\n\n```bash\ncrawlberg crawl https:\u002F\u002Fblog.unknown.example \\\n  --depth 2 --max-pages 50 --concurrent 4 --rate-limit 500 \\\n  --stay-on-domain --respect-robots-txt --format markdown\n```\n\nStay shallow, cap pages, throttle hard, obey robots.\n\n### Multi-seed batch\n\n```bash\ncrawlberg crawl \\\n  https:\u002F\u002Fexample.com\u002Fblog \\\n  https:\u002F\u002Fexample.com\u002Fdocs \\\n  https:\u002F\u002Fexample.com\u002Fpricing \\\n  --depth 2 --max-pages 100 --stay-on-domain --format json\n```\n\nJSON output for batch is an array of `{ seed_url, result }` entries — each\n`result` is a full crawl payload or `{ error: ... }`.\n\n## Output\n\n### Markdown mode\n\n```text\n---\nURL: https:\u002F\u002Fexample.com\u002Fpage-one\n---\n# Page One\n\n… markdown content …\n\n---\nURL: https:\u002F\u002Fexample.com\u002Fpage-two\n---\n…\n```\n\n### JSON mode\n\nTop-level `CrawlResult` with `pages: [...]`. Each page carries the rendered\nMarkdown plus metadata, links, images, JSON-LD, and HTTP response info. Read\n`result.pages[i].markdown.content` for the Markdown string.\n\n## Politeness checklist\n\n- Pass `--respect-robots-txt` on every third-party crawl.\n- Cap `--max-pages` — a runaway BFS can issue tens of thousands of requests.\n- Bump `--rate-limit` for hosts that show signs of stress (5xx, slowdowns).\n- Identify yourself via `--user-agent crawlberg (contact@example.com)`.\n\n## Common pitfalls\n\n- **No pages returned.** The seed page may be JS-only — the engine falls\n  back to headless automatically in `--browser-mode auto`, but `never` mode\n  will silently produce an empty crawl. Re-run with `--browser-mode always`\n  or check the headless-fallback skill.\n- **Crawl leaves the domain.** Pass `--stay-on-domain`. Combine with\n  `allow_subdomains: true` in `--config` JSON to include subdomains.\n- **Slow crawl.** The default rate limit is 200 ms per origin — multiple\n  seed URLs on the same host still share the bucket. Spread seeds across\n  hosts or raise `--concurrent` for unrelated origins.\n- **Memory growth.** Each page carries full Markdown plus structured data.\n  Stream JSON output to a file rather than holding it in memory; set\n  `--max-pages` aggressively if downstream cannot keep up.\n\n## When to reach for `map` instead\n\nIf the user only needs the list of URLs (sitemap analysis, link planning,\nseeding another tool), use `crawlberg map \u003Curl>` — it skips rendering and\nreturns a flat `MapResult` with hundreds of URLs in seconds.\n","\u003C!--\nAI-RULEZ :: GENERATED FILE — DO NOT EDIT\nContent-Hash: blake3:87e62f8235715c6d9ff820d40d26b521d5d0e0fb9888f7d04589c7429d28da14\nSource-Hash: blake3:b5689383a3914da8e3cc6ad3614356ae7f98c90cf99cb1b4b47be38456ff7a7c\nSchema-Version: v1\n-->\n\n# Crawling a site\n\nReach for `crawlberg crawl` when one URL is not enough — the user wants\nthe docs site, the blog, the marketing pages, or the whole domain.\n\n## Quick recipe\n\n```bash\ncrawlberg crawl https:\u002F\u002Fexample.com \\\n  --depth 3 \\\n  --max-pages 200 \\\n  --concurrent 8 \\\n  --rate-limit 250 \\\n  --stay-on-domain \\\n  --respect-robots-txt \\\n  --format markdown\n```\n\nDefaults you should usually override:\n\n- `--depth 2` is shallow — set it explicitly.\n- `--max-pages` is unbounded by default; cap it for any unknown site.\n- `--concurrent 10` is aggressive for small hosts; drop to 4-8 for\n  third-party sites.\n\n## Flag surface\n\n| Flag                    | Default | Purpose                                                    |\n| ----------------------- | ------- | ---------------------------------------------------------- |\n| `--depth`, `-d`         | `2`     | Maximum hop count from the seed URL.                       |\n| `--max-pages`, `-n`     | —       | Hard cap on pages fetched. Set this on any unknown site.   |\n| `--concurrent`, `-c`    | `10`    | Parallel in-flight requests.                               |\n| `--rate-limit`          | `200`   | Milliseconds between requests to the same origin.          |\n| `--stay-on-domain`      | off     | Skip links that leave the seed domain.                     |\n| `--respect-robots-txt`  | off     | Honour `robots.txt`. Pass it for any third-party host.     |\n| `--proxy`               | —       | HTTP, HTTPS, or SOCKS5 proxy URL.                          |\n| `--user-agent`          | —       | Override the request UA. Be honest.                        |\n| `--timeout`             | `30000` | Per-request timeout in ms.                                 |\n| `--browser-mode`        | `auto`  | `auto`, `always`, `never` — see the headless-fallback skill. |\n| `--browser-endpoint`    | —       | External CDP `ws:\u002F\u002F` URL.                                  |\n| `--format`              | `json`  | `json` or `markdown`.                                      |\n| `--config`              | —       | Inline JSON or `@file.json` for the full `CrawlConfig`.    |\n\nMultiple seed URLs are accepted positionally — the engine fans out with\n`batch_crawl` and aggregates results.\n\n## When to pick which flags\n\n### Docs sites you own\n\n```bash\ncrawlberg crawl https:\u002F\u002Fdocs.example.com \\\n  --depth 5 --max-pages 1000 --concurrent 16 --rate-limit 100 \\\n  --stay-on-domain --format markdown > docs.md\n```\n\nHigher concurrency and lower rate limits are fine on infrastructure you\ncontrol.\n\n### Third-party sites\n\n```bash\ncrawlberg crawl https:\u002F\u002Fblog.unknown.example \\\n  --depth 2 --max-pages 50 --concurrent 4 --rate-limit 500 \\\n  --stay-on-domain --respect-robots-txt --format markdown\n```\n\nStay shallow, cap pages, throttle hard, obey robots.\n\n### Multi-seed batch\n\n```bash\ncrawlberg crawl \\\n  https:\u002F\u002Fexample.com\u002Fblog \\\n  https:\u002F\u002Fexample.com\u002Fdocs \\\n  https:\u002F\u002Fexample.com\u002Fpricing \\\n  --depth 2 --max-pages 100 --stay-on-domain --format json\n```\n\nJSON output for batch is an array of `{ seed_url, result }` entries — each\n`result` is a full crawl payload or `{ error: ... }`.\n\n## Output\n\n### Markdown mode\n\n```text\n---\nURL: https:\u002F\u002Fexample.com\u002Fpage-one\n---\n# Page One\n\n… markdown content …\n\n---\nURL: https:\u002F\u002Fexample.com\u002Fpage-two\n---\n…\n```\n\n### JSON mode\n\nTop-level `CrawlResult` with `pages: [...]`. Each page carries the rendered\nMarkdown plus metadata, links, images, JSON-LD, and HTTP response info. Read\n`result.pages[i].markdown.content` for the Markdown string.\n\n## Politeness checklist\n\n- Pass `--respect-robots-txt` on every third-party crawl.\n- Cap `--max-pages` — a runaway BFS can issue tens of thousands of requests.\n- Bump `--rate-limit` for hosts that show signs of stress (5xx, slowdowns).\n- Identify yourself via `--user-agent crawlberg (contact@example.com)`.\n\n## Common pitfalls\n\n- **No pages returned.** The seed page may be JS-only — the engine falls\n  back to headless automatically in `--browser-mode auto`, but `never` mode\n  will silently produce an empty crawl. Re-run with `--browser-mode always`\n  or check the headless-fallback skill.\n- **Crawl leaves the domain.** Pass `--stay-on-domain`. Combine with\n  `allow_subdomains: true` in `--config` JSON to include subdomains.\n- **Slow crawl.** The default rate limit is 200 ms per origin — multiple\n  seed URLs on the same host still share the bucket. Spread seeds across\n  hosts or raise `--concurrent` for unrelated origins.\n- **Memory growth.** Each page carries full Markdown plus structured data.\n  Stream JSON output to a file rather than holding it in memory; set\n  `--max-pages` aggressively if downstream cannot keep up.\n\n## When to reach for `map` instead\n\nIf the user only needs the list of URLs (sitemap analysis, link planning,\nseeding another tool), use `crawlberg map \u003Curl>` — it skips rendering and\nreturns a flat `MapResult` with hundreds of URLs in seconds.\n",{"name":5,"description":7},"main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002Fkreuzberg-dev\u002Fplugins\u002Fplugins\u002Fcrawlberg\u002Fskills\u002Fcrawling-a-site",491,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-14 14:06:13",{"source":8,"skillName":5,"commands":57,"prompt":67,"zipAvailable":68},[58,61,64],{"tool":59,"command":60},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill crawling-a-site",{"tool":62,"command":63},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill crawling-a-site",{"tool":65,"command":66},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill crawling-a-site","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fkreuzberg-dev\u002Fplugins\u002Fplugins\u002Fcrawlberg\u002Fskills\u002Fcrawling-a-site\u002FSKILL.md",true,[],[71,79,86,93,119,143],{"skillNo":72,"name":73,"nameZh":6,"description":74,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":75,"categories":76,"keywords":77,"publishedAt":78},"01M746ZWACVKMZ1BJXJZHYJSSX","migrate-to-e2e","Moves an existing Playwright, Cypress, Selenium, Appium, Detox, Maestro, or TesterArmy suite to e2e one slice at a time, with the old suite running beside the new one until each ported test is proven. Keeps a MIGRATION.md progress file so any later session resumes where the last stopped. Optionally plants a defect to show both suites catch it. Use for \u002Fmigrate-to-e2e, \"migrate our tests to e2e\", \"port these Maestro flows\", \"move our TesterArmy tests\", or when a project has an e2e config and an older suite still running.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Ftester-army\u002Fe2e\u002Fskills\u002Fmigrate-to-e2e\u002FSKILL.md",[],[],"2026-11-11 15:11:14",{"skillNo":80,"name":81,"nameZh":6,"description":82,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":78},"01M746ZPTAX1Q0FBKAB1CYND7E","e2e","Writes, runs, and debugs end-to-end tests for web and mobile apps with e2e, the agentic test runner, which mixes agent.act goals with exact locator assertions on a Playwright browser or an iOS or Android device. Also drives a project's running app live over the e2e MCP server to look at a screen, check a UI change, take a screenshot or record a video, and try a locator; and runs bug bashes, parallel e2e explore runs with each bug proven by a repro test. Use when a project depends on e2e or has an e2e.config.ts, when asked for end-to-end, browser, mobile, or agentic UI tests, to check a change in the running app, to QA or bug bash an app or branch, or when an e2e run fails.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Ftester-army\u002Fe2e\u002Fskills\u002Fe2e\u002FSKILL.md",[],[],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":78},"01M746ZJN784HVHMDP9MQGCSGG","create-verification-skill","Generate a project-local verify-\u003Capp> skill on top of e2e, so any coding agent can launch the app, check it is worth driving, exercise a feature the way a user does, keep evidence, and bug bash it. Writes the skill, a feature map with one explore charter per feature, and proves it once. Use for \u002Fcreate-verification-skill, \"make a verification skill for this repo\", or when a project has no scripted way to prove UI behavior.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Ftester-army\u002Fe2e\u002Fskills\u002Fcreate-verification-skill\u002FSKILL.md",[],[],{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":41},"01M716A6ETNB9B1SQ3GXZC75VR","deja-search","deja-vu memory — search the user's past AI coding sessions with the deja CLI. Holds what they already settled and never wrote into the repo (naming rules, banned or already-tried tools, required checks before merge, preferred values, who owns what). Use before your first edit in a task, before you call a change done or ready to merge, before you recommend a tool, library, design or reviewer, before re-debugging an error that may already be solved, when they say 'didn't we fix this before', 'what did we decide about X' or 'I already have that', and before telling them something on this machine does not exist.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fvshulcz\u002Fdeja-vu\u002Fskills\u002Fdeja-search\u002FSKILL.md",[],[100,101,102,103,104,105,106,107,108,109,110,23,111,112,113,114,115,116,117,118],"search","deja search","never","error","ready","memory","re","decide","fix","design","debugging","you","done","reviewer","io","change","cli","merge","past",{"skillNo":120,"name":121,"nameZh":6,"description":122,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":123,"categories":124,"keywords":125,"publishedAt":41},"01M716A3J2W5SENSE9VGMXB375","deja-reflect","End-of-session reflection and memory curation with deja-vu. Review what was decided, what commands broke and what repaired them, and record durable project rules or decisions with deja remember so the next session starts with them.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fvshulcz\u002Fdeja-vu\u002Fskills\u002Fdeja-reflect\u002FSKILL.md",[],[126,127,128,114,28,129,130,131,132,133,134,135,136,137,105,138,139,140,141,142],"deja reflect","reflect","review","record","github","next","https","node","Rules","decisions","brew","remember","project rules","project","session","end","reflection",{"skillNo":144,"name":145,"nameZh":6,"description":146,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":147,"categories":148,"keywords":160,"publishedAt":41},"01M715T3RR4BAYSR57HPPTJ0PF","revid","Make, edit, export and post AI videos with Revid (revid.ai) through the Revid MCP server. Use whenever the user wants a short-form or long-form video made (from an idea, a script, a URL, an article, a PDF, a tweet, a product page, their own clips, or a song), wants to find what is going viral on TikTok, YouTube, Instagram or X before making something, wants to edit or re-export an existing Revid project, generate a standalone AI image or clip, make a series episode, or publish and schedule a finished video to connected social accounts. Covers setup (hosted MCP with OAuth, or an API key), the 36 tools, credits, polling and the rules for publishing. Routes to the more specific revid-* skills when one fits.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Ftmakersas\u002Frevid-skills\u002Fskills\u002Frevid\u002FSKILL.md",[149,154,157],{"categoryNo":150,"categoryName":151,"taxonomyType":152,"taxonomyTypeDesc":153},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",{"categoryNo":155,"categoryName":156,"taxonomyType":152,"taxonomyTypeDesc":153},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":158,"categoryName":159,"taxonomyType":152,"taxonomyTypeDesc":153},"01JBSEEDCAT000000000000010","后端与接口",[145,161,162,163,164,165,166,139,167,106,168,169,170,171,23,172,173,174,175,176],"URL","Video","short-form","standalone","PDF","viral","tiktok","short","social","MCP","song","image","mcp server","schedule","api","publish"]