brightdata-ci-integration
Configure Bright Data CI/CD integration with GitHub Actions and testing. Use when setting up automated testing, configuring CI pipelines, or integrating Bright Data tests into your build process. Trigger with phrases like "brightdata CI", "brightdata GitHub Actions", "brightdata automated tests", "CI brightdata".
- License 许可
- MIT
- 最近更新
- 2026-07-13
- Stars
- ★ 1
- Forks
- 0
01 前言 Front Matter
- name
- brightdata-ci-integration
- tags
- [ "saas", "scraping", "data", "brightdata" ]
- author
- Jeremy Longshore <jeremy@intentsolutions.io>
- license
- MIT
- version
- 1.0.0
- description
- Configure Bright Data CI/CD integration with GitHub Actions and testing. Use when setting up automated testing, configuring CI pipelines, or integrating Bright Data tests into your build process. Trigger with phrases like "brightdata CI", "brightdata GitHub Actions", "brightdata automated tests", "CI brightdata".
- allowed-tools
- Read, Write, Edit, Bash(gh:*)
- compatible-with
- claude-code
03 SKILL.md 正文
Bright Data CI Integration
Overview
Set up CI/CD pipelines for Bright Data scraping projects with GitHub Actions. Includes mocked unit tests that run without proxy access and optional live integration tests that verify actual proxy connectivity.
Prerequisites
- GitHub repository with Actions enabled
- Bright Data test zone credentials
- npm/pnpm project configured
Instructions
Step 1: GitHub Actions Workflow
# .github/workflows/scraper-tests.yml
name: Scraper Tests
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
unit-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- run: npm ci
- run: npm test -- --coverage
# Unit tests use mocked proxy responses — no credentials needed
integration-tests:
runs-on: ubuntu-latest
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
env:
BRIGHTDATA_CUSTOMER_ID: ${{ secrets.BRIGHTDATA_CUSTOMER_ID }}
BRIGHTDATA_ZONE: ${{ secrets.BRIGHTDATA_ZONE }}
BRIGHTDATA_ZONE_PASSWORD: ${{ secrets.BRIGHTDATA_ZONE_PASSWORD }}
BRIGHTDATA_API_TOKEN: ${{ secrets.BRIGHTDATA_API_TOKEN }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- run: npm ci
- name: Download Bright Data CA cert
run: curl -sO https://brightdata.com/ssl/brd-ca.crt
- name: Verify proxy connectivity
run: |
curl -x "http://brd-customer-${BRIGHTDATA_CUSTOMER_ID}-zone-${BRIGHTDATA_ZONE}:${BRIGHTDATA_ZONE_PASSWORD}@brd.superproxy.io:33335" \
-s https://lumtest.com/myip.json | python3 -m json.tool
- run: npm run test:integration
Step 2: Configure GitHub Secrets
gh secret set BRIGHTDATA_CUSTOMER_ID --body "c_abc123"
gh secret set BRIGHTDATA_ZONE --body "web_unlocker_test"
gh secret set BRIGHTDATA_ZONE_PASSWORD --body "z_test_password"
gh secret set BRIGHTDATA_API_TOKEN --body "test_api_token"
Step 3: Write Mocked Unit Tests
// tests/unit/scraper.test.ts — runs without Bright Data credentials
import { describe, it, expect, vi, beforeEach } from 'vitest';
import axios from 'axios';
vi.mock('axios');
describe('Scraper', () => {
beforeEach(() => vi.clearAllMocks());
it('should configure proxy correctly', async () => {
vi.mocked(axios.create).mockReturnValue({
get: vi.fn().mockResolvedValue({ status: 200, data: '<html>OK</html>' }),
} as any);
const { getBrightDataClient } = await import('../../src/brightdata/client');
const client = getBrightDataClient();
expect(axios.create).toHaveBeenCalledWith(
expect.objectContaining({
proxy: expect.objectContaining({ host: 'brd.superproxy.io', port: 33335 }),
})
);
});
it('should parse HTML response into structured data', async () => {
const { parseProductPage } = await import('../../src/brightdata/parser');
const result = parseProductPage('<html><h1>Product</h1><span class="price">$29.99</span></html>');
expect(result.title).toBe('Product');
expect(result.price).toBe('$29.99');
});
});
Step 4: Write Live Integration Tests
// tests/integration/proxy.test.ts
import { describe, it, expect } from 'vitest';
const LIVE = process.env.BRIGHTDATA_CUSTOMER_ID && process.env.BRIGHTDATA_ZONE;
describe.skipIf(!LIVE)('Bright Data Live Integration', () => {
it('should connect through proxy', async () => {
const { getBrightDataClient } = await import('../../src/brightdata/client');
const client = getBrightDataClient();
const res = await client.get('https://lumtest.com/myip.json');
expect(res.status).toBe(200);
expect(res.data).toHaveProperty('ip');
expect(res.data).toHaveProperty('country');
}, 30000);
it('should scrape through Web Unlocker', async () => {
const { getBrightDataClient } = await import('../../src/brightdata/client');
const client = getBrightDataClient();
const res = await client.get('https://example.com');
expect(res.status).toBe(200);
expect(res.data).toContain('Example Domain');
}, 60000);
});
Output
- Unit tests run on every PR without proxy credentials
- Integration tests run on main push with live proxy
- GitHub secrets configured securely
- CA certificate downloaded in CI
Error Handling
| Issue | Cause | Solution |
|---|---|---|
| Integration tests fail | Missing secrets | Add via gh secret set |
| Proxy timeout in CI | Slow CAPTCHA | Increase test timeout to 60s |
| Flaky tests | IP rotation variability | Use lumtest.com for stable verification |
Resources
Next Steps
For deployment patterns, see brightdata-deploy-integration.
04 同仓库更多 Skills
zustand-store-creator
Create zustand store creator operations. Auto-activating skill for Frontend Development. Triggers on: zustand store creator, zustand store creator Part of the Frontend Development skill category. Use when working with zustand store creator functionality. Trigger with phrases like "zustand store creator", "zustand creator", "zustand".
zero-trust-config-helper
Configure with zero trust config helper operations. Auto-activating skill for Security Advanced. Triggers on: zero trust config helper, zero trust config helper Part of the Security Advanced skill category. Use when configuring systems or services. Trigger with phrases like "zero trust config helper", "zero helper", "zero".
zapier-integration-helper
Assist with zapier integration helper operations. Auto-activating skill for Business Automation. Triggers on: zapier integration helper, zapier integration helper Part of the Business Automation skill category. Use when working with APIs or building integrations. Trigger with phrases like "zapier integration helper", "zapier helper", "zapier".
yaml-config-validator
Validate yaml config validator operations. Auto-activating skill for DevOps Basics. Triggers on: yaml config validator, yaml config validator Part of the DevOps Basics skill category. Use when configuring systems or services. Trigger with phrases like "yaml config validator", "yaml validator", "yaml".
xvary-stock-research
Thesis-driven equity analysis from public SEC EDGAR and market data; /analyze, /score, /compare workflows with bundled Python tools (Claude Code, Cursor, Codex).
xss-vulnerability-scanner
Scan xss vulnerability scanner operations. Auto-activating skill for Security Fundamentals. Triggers on: xss vulnerability scanner, xss vulnerability scanner Part of the Security Fundamentals skill category. Use when working with xss vulnerability scanner functionality. Trigger with phrases like "xss vulnerability scanner", "xss scanner", "xss".