bigquery-data-transfer-service
Discovers and inspects BigQuery Data Transfer Service (DTS) configurations. Use this to identify existing ingestion pipelines and extract datasource or transfer config metadata for data pipelines. Use when a user asks for ingestion scenarios while building or managing data pipelines or when a user asks to "ingest" or "add" data that may already be managed by a DTS transfer.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-21
- Stars
- ★ 3
- Forks
- 0
01 前言 Front Matter
- name
- bigquery-data-transfer-service
- license
- Apache-2.0
- metadata
- { "version": "v1", "publisher": "google" }
- description
- Discovers and inspects BigQuery Data Transfer Service (DTS) configurations. Use this to identify existing ingestion pipelines and extract datasource or transfer config metadata for data pipelines. Use when a user asks for ingestion scenarios while building or managing data pipelines or when a user asks to "ingest" or "add" data that may already be managed by a DTS transfer.
02 资源文件 1
- 脚本scripts/bigquery_dts.py4.9 KB
03 SKILL.md 正文
BigQuery Data Transfer Service (DTS)
Mandatory Guidelines
[!IMPORTANT]
All new BigQuery Data Transfer Service (DTS) configurations MUST be provisioned through the gcp pipeline resource provisioning framework, which includes generating a
deployment.yaml.
- Do NOT use imperative CLI commands (e.g.,
bq mkorgcloud) to create or update configurations.- CLI commands are permitted only for discovery (listing/showing) and triggering manual runs.
This guide enables the discovery of existing ingestion resources and provides metadata related to ingestion when needed.
Workflow
Step 0: Discover Environment Parameters
Before generating configurations, discover the actual values for the target project and region.
[!TIP]
If
deployment.yamlalready exists in the repository root, prioritize extractingprojectandregionfrom the target environment configuration (e.g.,dev).
- Project:
gcloud config get project - Region:
gcloud config get compute/region
[!TIP]
Use these commands to replace placeholders like
<PROJECT_ID>with actual values. Always remove associated comments that start with TODO once replaced.
Step 1: Check for Existing Transfers
Before assuming a new transfer is needed, check for existing ones in the target region.
-
List Transfers:
bq ls --transfer_config \ --transfer_location=<REGION> \ --project_id=<PROJECT_ID> -
Analyze Existing Transfers:
-
Single Transfer Found:
- Check if the transfer has at least one successful run:
bq ls --transfer_run --transfer_config=<RESOURCE_NAME> - If found: Use existing transfer config.
- If not found: Confirm with user if it's ok to trigger the transfer run.
- Check if the transfer has at least one successful run:
-
Multiple Transfers Found:
- Attempt to guess the correct one based on context.
- Ask user to confirm.
-
Disabled Transfers Found:
- Ask user if they want to enable it or create a new one.
- To Enable: Instruct the user to update the transfer configuration
within their
deployment.yamlfile by setting thedisabledfield tofalsefor the specific transfer resource.
-
No Transfers Found: Proceed to create new if needed.
-
Step 2: Discover & Validate Parameters (New Transfers)
If creating a new transfer, discover the required parameters using the REST API and validate them with the user.
[!TIP]
If
<DATA_SOURCE_ID>is unknown, run the discovery script without<DATA_SOURCE_ID>argument to list available source IDs (e.g.,google_cloud_storage). It uses the derived project and location from Step 0.
python3 scripts/bigquery_dts.py --project_id=<PROJECT_ID>
-
Run Discovery Script: Use the
bigquery_dts.pyscript to inspect Data Source parameters via the REST API.# Passes the derived project and region to the script. python3 scripts/bigquery_dts.py --project_id=<PROJECT_ID> <DATA_SOURCE_ID> <REGION>[!IMPORTANT]
Run this command every time a new transfer is being planned.
-
[!CAUTION]
Mandatory User Questionnaire (CRITICAL):
- Explicitly identify ALL specific parameters returned by the discovery script. You MUST NOT generalize or vaguely summarize them.
- OAuth Authorization (Google Data Sources): For Google ecosystem data
sources (Google Ads, Youtube, etc.), if the user is not using a service
account to configure the DTS transfer config (meaning the user is using
End User Credentials or EUC to configure the transfer config), then
generate an OAuth URI. Ask the user to visit this URL to authorize. Once
the user provides the versionInfo code, use the code as
definition.versionInfoindeployment.yamland then you can proceed. - If any parameters are related to authentication, explicitly ask the user to provide the Secret Manager Resource ID (e.g., projects/my-project/secrets/my-secret) for these parameters
- Present every required parameter to the user BEFORE generating config files.
- Ask for verification of assets/tables to be ingested.
-
Wait for User Response: You MUST NOT proceed until parameters are confirmed.
Step 3: Extract Transfer Config Data
Retrieve the configuration details for the selected transfer.
bq show --format=prettyjson --transfer_config <RESOURCE_NAME>
Step 4: Trigger and Verify Transfer
After the transfer is deployed via the resource provisioning framework, you MUST ensure there is at least a single successful run before proceeding with the rest of the tasks.
-
Trigger a Manual Run: If no successful runs or ongoing runs are found, or the transfer was just created, trigger a manual run for the current time.
bq mk --transfer_run \ --transfer_config=<RESOURCE_NAME> \ --run_time=$(date -u +"%Y-%m-%dT%H:%M:%SZ") -
Poll for Completion (5-Minute Rule): Attempt to check the status of the run every 30-60 seconds for up to 5 minutes.
bq ls --format=prettyjson --transfer_run --transfer_config=<RESOURCE_NAME>- Success: If the run completes successfully, proceed with the rest of the pipeline.
- Failure: If the run fails, analyze the logs and ask the user for help.
- Timeout (5 mins): If the run is still in progress after 5 minutes, STOP and ask the user: "The Data Transfer Service ingestion is still in progress. Please provide 'proceed guidance' once the ingestion has finished so that I can continue building the rest of the data pipeline using the ingested schema and samples."
-
Wait for User Guidance: Do NOT proceed until the user confirms ingestion is complete or provides guidance.
-
Once user confirms to proceed, start work on rest of the tasks.
Definition of Done
- A BigQuery DTS transfer configuration has been discovered or provisioned
declaratively (via gcp pipeline resource provisioning with a generated
deployment.yaml). - Mandatory datasource parameters have been identified and confirmed with the user.
- A manual transfer run has been triggered and monitored.
- The transfer run has completed successfully OR the user has provided "proceed guidance" for a long-running transfer.
04 同仓库更多 Skills
webapp-testing
Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.
web-artifacts-builder
Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn/ui). Use for complex artifacts requiring state management, routing, or shadcn/ui components - not for simple single-file HTML/JSX artifacts.
ui-styling
Create beautiful, accessible user interfaces with shadcn/ui components (built on Radix UI + Tailwind), Tailwind CSS utility-first styling, and canvas-based visual designs. Use when building user interfaces, implementing design systems, creating responsive layouts, adding accessible components (dialogs, dropdowns, forms, tables), customizing themes and colors, implementing dark mode, generating visual designs and posters, or establishing consistent styling patterns across applications.
design
Comprehensive design skill: brand identity, design tokens, UI styling, logo generation (55 styles, Gemini AI), corporate identity program (50 deliverables, CIP mockups), HTML presentations (Chart.js), banner design (22 styles, social/ads/web/print), icon design (15 styles, SVG, Gemini 3.1 Pro), social photos (HTML→screenshot, multi-platform). Actions: design logo, create CIP, generate mockups, build slides, design banner, generate icon, create social photos, social media images, brand identity, design system. Platforms: Facebook, Twitter, LinkedIn, YouTube, Instagram, Pinterest, TikTok, Threads, Google Ads.
design-system
Token architecture, component specifications, and slide generation. Three-layer tokens (primitive→semantic→component), CSS variables, spacing/typography scales, component specs, strategic slide creation. Use for design tokens, systematic design, brand-compliant presentations.
banner-design
Design banners for social media, ads, website heroes, creative assets, and print. Multiple art direction options with AI-generated visuals. Actions: design, create, generate banner. Platforms: Facebook, Twitter/X, LinkedIn, YouTube, Instagram, Google Display, website hero, print. Styles: minimalist, gradient, bold typography, photo-based, illustrated, geometric, retro, glassmorphism, 3D, neon, duotone, editorial, collage. Uses ui-ux-pro-max, frontend-design, ai-artist, ai-multimodal skills.