data-feeds
Extract structured data from 40+ websites including Amazon, LinkedIn, Instagram, TikTok, Facebook, YouTube, and more. Uses Bright Data's Web Data APIs with automatic polling. Returns clean JSON with product details, profiles, reviews, posts, and comments.
- License 许可
- MIT
- 最近更新
- 2026-08-16
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- data-feeds
- description
- Extract structured data from 40+ websites including Amazon, LinkedIn, Instagram, TikTok, Facebook, YouTube, and more. Uses Bright Data's Web Data APIs with automatic polling. Returns clean JSON with product details, profiles, reviews, posts, and comments.
02 资源文件 2
- 脚本scripts/datasets.sh8.4 KB
- 脚本scripts/fetch.sh1.9 KB
03 SKILL.md 正文
Bright Data - Structured Data Feeds
Extract structured data from major websites with automatic parsing. No scraping logic needed - just provide a URL and get clean JSON data.
Setup
Environment Variables (Required)
export BRIGHTDATA_API_KEY="your-api-key"
Optional
export BRIGHTDATA_POLLING_TIMEOUT=600 # Max seconds to wait (default: 600)
Get your API key from Bright Data Dashboard.
Usage
bash scripts/datasets.sh <dataset_type> <url> [additional_params...]
Available Datasets
E-Commerce
| Dataset | Command | Description |
|---|---|---|
| Amazon Product | datasets.sh amazon_product <url> |
Product details, pricing, ratings |
| Amazon Reviews | datasets.sh amazon_product_reviews <url> |
Customer reviews for a product |
| Amazon Search | datasets.sh amazon_product_search <keyword> <domain_url> |
Search results |
| Walmart Product | datasets.sh walmart_product <url> |
Product details from Walmart |
| Walmart Seller | datasets.sh walmart_seller <url> |
Seller information |
| eBay Product | datasets.sh ebay_product <url> |
eBay listing details |
| Home Depot | datasets.sh homedepot_products <url> |
Home Depot product data |
| Zara | datasets.sh zara_products <url> |
Zara product details |
| Etsy | datasets.sh etsy_products <url> |
Etsy listing data |
| Best Buy | datasets.sh bestbuy_products <url> |
Best Buy product info |
Professional Networks
| Dataset | Command | Description |
|---|---|---|
| LinkedIn Person | datasets.sh linkedin_person_profile <url> |
Profile data (experience, skills) |
| LinkedIn Company | datasets.sh linkedin_company_profile <url> |
Company page data |
| LinkedIn Jobs | datasets.sh linkedin_job_listings <url> |
Job posting details |
| LinkedIn Posts | datasets.sh linkedin_posts <url> |
Post content and engagement |
| LinkedIn Search | datasets.sh linkedin_people_search <url> <first> <last> |
Find people |
| Crunchbase | datasets.sh crunchbase_company <url> |
Company funding, employees |
| ZoomInfo | datasets.sh zoominfo_company_profile <url> |
Company profile data |
| Dataset | Command | Description |
|---|---|---|
| Profiles | datasets.sh instagram_profiles <url> |
Bio, followers, following |
| Posts | datasets.sh instagram_posts <url> |
Post details, likes, captions |
| Reels | datasets.sh instagram_reels <url> |
Reel data and metrics |
| Comments | datasets.sh instagram_comments <url> |
Post comments |
| Dataset | Command | Description |
|---|---|---|
| Posts | datasets.sh facebook_posts <url> |
Post content and reactions |
| Marketplace | datasets.sh facebook_marketplace_listings <url> |
Listing details |
| Reviews | datasets.sh facebook_company_reviews <url> [num] |
Company reviews |
| Events | datasets.sh facebook_events <url> |
Event details |
TikTok
| Dataset | Command | Description |
|---|---|---|
| Profiles | datasets.sh tiktok_profiles <url> |
Creator profile data |
| Posts | datasets.sh tiktok_posts <url> |
Video details and metrics |
| Shop | datasets.sh tiktok_shop <url> |
TikTok Shop product data |
| Comments | datasets.sh tiktok_comments <url> |
Video comments |
YouTube
| Dataset | Command | Description |
|---|---|---|
| Profiles | datasets.sh youtube_profiles <url> |
Channel data |
| Videos | datasets.sh youtube_videos <url> |
Video details and stats |
| Comments | datasets.sh youtube_comments <url> [num] |
Video comments (default: 10) |
Other Social
| Dataset | Command | Description |
|---|---|---|
| X (Twitter) | datasets.sh x_posts <url> |
Tweet data |
datasets.sh reddit_posts <url> |
Post and comment data |
Google Services
| Dataset | Command | Description |
|---|---|---|
| Maps Reviews | datasets.sh google_maps_reviews <url> [days] |
Business reviews (default: 3 days) |
| Shopping | datasets.sh google_shopping <url> |
Product comparison data |
| Play Store | datasets.sh google_play_store <url> |
App details and reviews |
Other
| Dataset | Command | Description |
|---|---|---|
| Apple App Store | datasets.sh apple_app_store <url> |
iOS app data |
| Reuters News | datasets.sh reuter_news <url> |
News article content |
| GitHub | datasets.sh github_repository_file <url> |
Repository file data |
| Yahoo Finance | datasets.sh yahoo_finance_business <url> |
Stock and company data |
| Zillow | datasets.sh zillow_properties_listing <url> |
Property listing details |
| Booking.com | datasets.sh booking_hotel_listings <url> |
Hotel listing data |
Examples
Get LinkedIn Profile
bash scripts/datasets.sh linkedin_person_profile "https://www.linkedin.com/in/satyanadella/"
Get Amazon Product
bash scripts/datasets.sh amazon_product "https://www.amazon.com/dp/B09V3KXJPB"
Get Instagram Profile
bash scripts/datasets.sh instagram_profiles "https://www.instagram.com/natgeo/"
Get YouTube Comments
bash scripts/datasets.sh youtube_comments "https://www.youtube.com/watch?v=dQw4w9WgXcQ" 20
Search Amazon
bash scripts/datasets.sh amazon_product_search "wireless headphones" "https://www.amazon.com"
Output Format
Returns structured JSON with website-specific fields. Example for LinkedIn profile:
{
"name": "Satya Nadella",
"headline": "Chairman and CEO at Microsoft",
"location": "Greater Seattle Area",
"connections": "500+",
"experience": [...],
"education": [...],
"skills": [...]
}
How It Works
- Trigger: Sends URL to Bright Data's Web Data API
- Poll: Waits for data collection to complete (checks every second)
- Return: Outputs structured JSON when ready
The polling mechanism handles rate limits and ensures data quality by waiting for full extraction.
Advanced: Direct Fetch
For custom dataset IDs or advanced use cases:
bash scripts/fetch.sh <dataset_id> '<json_input>'
Example:
bash scripts/fetch.sh gd_l1viktl72bvl7bjuj0 '{"url":"https://linkedin.com/in/someone"}'
04 同仓库更多 Skills
zapier-make-patterns
No-code automation democratizes workflow building. Zapier and Make (formerly Integromat) let non-developers automate business processes without writing code. But no-code doesn't mean no-complexity - these platforms have their own patterns, pitfalls, and breaking points. This skill covers when to use which platform, how to build reliable automations, and when to graduate to code-based solutions. Key insight: Zapier optimizes for simplicity and integrations (7000+ apps), Make optimizes for power
yeet
Use only when the user explicitly asks to stage, commit, push, and open a GitHub pull request in one flow using the GitHub CLI (`gh`).
workflow-automation
Workflow automation is the infrastructure that makes AI agents reliable. Without durable execution, a network hiccup during a 10-step payment flow means lost money and angry customers. With it, workflows resume exactly where they left off. This skill covers the platforms (n8n, Temporal, Inngest) and patterns (sequential, parallel, orchestrator-worker) that turn brittle scripts into production-grade automation. Key insight: The platforms make different tradeoffs. n8n optimizes for accessibility
trigger-dev
Trigger.dev expert for background jobs, AI workflows, and reliable async execution with excellent developer experience and TypeScript-first design. Use when: trigger.dev, trigger dev, background task, ai background job, long running task.
slack-automation
Automate Slack workspace operations including messaging, search, channel management, and reaction workflows through Composio's Slack toolkit.
planning-with-files
Transforms workflow to use Manus-style persistent markdown files for planning, progress tracking, and knowledge storage. Use when starting complex tasks, multi-step projects, research tasks, or when the user mentions planning, organizing work, tracking progress, or wants structured output.