kirk-deep-scraper
暂无简介
- License 许可
- MIT
- 最近更新
- 2026-06-10
- Stars
- ★ 3
- Forks
- 0
03 SKILL.md 正文
Skill: deep-scraper
Overview
A high-performance engineering tool for deep web scraping. It uses a containerized Docker + Crawlee (Playwright) environment to penetrate protections on complex websites like YouTube and X/Twitter, providing "interception-level" raw data.
Requirements
- Docker: Must be installed and running on the host machine.
- Image: Build the environment with the tag
skillboss-crawlee.- Build command:
docker build -t skillboss-crawlee skills/deep-scraper/
- Build command:
Integration Guide
Simply copy the skills/deep-scraper directory into your skills/ folder. Ensure the Dockerfile remains within the skill directory for self-contained deployment.
Standard Interface (CLI)
docker run -t --rm -v $(pwd)/skills/deep-scraper/assets:/usr/src/app/assets skillboss-crawlee node assets/main_handler.js [TARGET_URL]
Output Specification (JSON)
The scraping results are printed to stdout as a JSON string:
status: SUCCESS | PARTIAL | ERRORtype: TRANSCRIPT | DESCRIPTION | GENERICvideoId: (For YouTube) The validated Video ID.data: The core text content or transcript.
Core Rules
- ID Validation: All YouTube tasks MUST verify the Video ID to prevent cache contamination.
- Privacy: Strictly forbidden from scraping password-protected or non-public personal information.
- Alpha-Focused: Automatically strips ads and noise, delivering pure data optimized for LLM processing.
04 同仓库更多 Skills
youtube-content
YouTube transcripts to summaries, threads, blogs.
xurl
X/Twitter via xurl CLI: post, search, DM, media, v2 API.
touchdesigner-mcp
Control a running TouchDesigner instance via twozero MCP — create operators, set parameters, wire connections, execute Python, build real-time visuals. 36 native tools.
test-driven-development
TDD: enforce RED-GREEN-REFACTOR, tests before code.
teams-meeting-pipeline
Operate the Teams meeting summary pipeline via Hermes CLI — summarize meetings, inspect pipeline status, replay jobs, manage Microsoft Graph subscriptions.
systematic-debugging
4-phase root cause debugging: understand bugs before fixing.