[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7BVBHNJFN402ZP6EZ6SE2YF":3,"skill-same:01M7BVBHNJFN402ZP6EZ6SE2YF":60},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":26,"commitSha":27,"skillDirectory":28,"forksCount":29,"repositoryHtmlUrl":30,"repositoryHomepageUrl":31,"ownerHtmlUrl":32,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":33,"install":34,"resources":47},"01M7BVBHNJFN402ZP6EZ6SE2YF","gcp-pipeline-orchestration",null,"This skill helps the agent generate or update orchestration pipeline definitions for Google Cloud Composer to initialize orchestration pipeline or update the orchestration definition for orchestration of various data pipelines, like dbt pipelines, notebooks, Spark jobs, Dataform, Python scripts or inline BigQuery SQL queries. This skill also helps deploy and trigger orchestration pipelines.","nguyenhungtran18\u002Fskill-and-tool-tracker","nguyenhungtran18","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F297063550?v=4","01M7BV0BFPRRFP84F2ME0FYZ9D","01M7BV0BG83FA5P22NJV0GR0A5",1,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fnguyenhungtran18\u002Fskill-and-tool-tracker\u002Fblob\u002Fmain\u002Fskills\u002Fgcp_pipeline_orchestration\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: gcp-pipeline-orchestration\ndescription: This skill helps the agent generate or update orchestration pipeline\n  definitions for Google Cloud Composer to initialize orchestration pipeline or update\n  the orchestration definition for orchestration of various data pipelines, like dbt\n  pipelines, notebooks, Spark jobs, Dataform, Python scripts or inline BigQuery SQL\n  queries. This skill also helps deploy and trigger orchestration pipelines.\nlicense: Apache-2.0\nmetadata:\n  version: v1\n  publisher: google\n---\n\n## Mandatory Reference Routing\n\nIf relevant, call the associated reference file(s) before you take actions.\nRefer to the table below to determine which reference file to retrieve in\ndifferent scenarios involving specific functions. [!IMPORTANT]: DO NOT GUESS\nfilenames. You MUST only use the exact paths provided below.\n\nFunction\u002FUse Case                  | Required Reference File                        | Capabilities & Intent Keywords\n---------------------------------- | ---------------------------------------------- | ------------------------------\n**orchestration-pipelines schema** | `references\u002Forchestration-pipelines-schema.md` | orchestrate, generate, create, update\n\n## How to use this skill\n\nOrchestration pipelines require creating two files to ensure a complete and\ndeployable pipeline:\n\n1.  `Orchestration File` (e.g., `orchestration-pipeline.yaml`,\n    `test-pipeline.yaml`): Defines the pipeline's logic, tasks, and schedule.\n    **IMPORTANT:** Check if a `deployment.yaml` file exists and references an\n    existing orchestration file. If it does, you **must update the existing\n    orchestration file** (e.g.,`test_pipeline.yaml`) instead of creating a new\n    one. The filename can be customized but must be referenced in the\n    `deployment.yaml` file.\n2.  `deployment.yaml`: Defines the environment-specific configurations.(e.g.,\n    `dev`, `prod`). `deployment.yaml`should only exists in the repository root\n    and must be named `deployment.yaml`\n\n-   All files should always be maintained together. And all files should be\n    placed on the root of the workspace folder.\n\n-   This skill is helpful to create or update configuration files to orchestrate\n    data pipelines.\n\n## How to use this skill\n\n### Step 1: Assess Orchestration Pipeline Status and Initialize if Necessary\n\nExamine the repository's root directory for a `deployment.yaml` file.\n\n1.  **Check for existing setup**: The absence of `deployment.yaml` indicates\n    that orchestration has not been set up.\n2.  **Determine if initialization is required**: Initialization is required if\n    `deployment.yaml` is missing. you **MUST** run the `init` command in Step 3\n    to scaffold the project if `deployment.yaml` is missing. Do NOT create the\n    files manually.\n3.  **Pipeline Name**: If initialization is needed, ask the user for the\n    pipeline name. If user hasn't provided the orchestration pipeline name, name\n    should be \"orchestration_pipeline\"\n4.  **Environment Name**: If initialization is needed, you MUST ask the user for\n    the environment name. If the user does not provide it, use **dev** as the\n    default.\n\n5.  **Execute Initialization**: Once you have the pipeline name, run the\n    following command:\n\n```\n# Replace \u003CORCHESTRATION_PIPELINE_NAME> with the actual name\n# Replace \u003CENV_NAME> with the actual environment name\ngcloud beta orchestration-pipelines init \u003CORCHESTRATION_PIPELINE_NAME> --environment=\u003CENV_NAME>\n```\n\n### Step 2: Review the orchestration pipeline code structure and syntax instruction\n\n*** Pipeline Models (mapping to YAML)\n\n> [!IMPORTANT]\n>\n> While the internal pipeline models are defined using protobuf (which typically\n> uses `snake_case`), the **YAML configuration expects `camelCase`** for almost\n> all field names.\n>\n> **Mapping Rule:** Always convert `snake_case` proto fields (e.g.,\n> `pipeline_id`) to `camelCase` in YAML (e.g., `pipelineId`).\n\n#### Orchestration-Pipelines yaml structure and syntax instruction\n\nReference to file `references\u002Forchestration-pipelines-schema.md`.\n\n**Required Tags (Top-Level)**: You **MUST** add a `tags` field to the top-level\nof the orchestration pipeline YAML definition. The value of this field depends\non the IDE environment:\n\n-   For Antigravity, use `[\"job:datacloud:antigravity\"]`.\n-   For VS Code, use `[\"job:datacloud:vscode\"]`.\n-   For Gemini CLI, use `[\"job:datacloud:gemini-cli\"]`.\n-   For Claude Code, use `[\"job:datacloud:claude\"]`.\n-   For Codex, use `[\"job:datacloud:codex\"]`.\n-   For any other environment, use `[\"job:datacloud:other\"]`.\n\n#### Deployment yaml structure and syntax instruction.\n\n**Top-Level Structure:** The root of the YAML should be an object with the\nfollowing fields:\n\n-   `environments` (dictionary): A map where keys are environment names (e.g.,\n    'dev', 'prod', etc) and values are Environment objects.\n\n**Environment:** Each environment object contains the following fields:\n\n-   `project` (string): The Google Cloud Project ID.\n-   `region` (string): The Google Cloud region (e.g., 'us-central1').\n-   `composer_environment` (string): The Cloud Composer environment name.\n-   `artifact_storage`\n    -   `bucket` (string): GCS bucket\n    -   `path_prefix`(string): prefix of path that we want to put in bucket\n-   `pipelines`\n    -   `- source` (string): orchestration pipeline yaml file names. It can be\n        multiple\n-   `variables` (dictionary, optional): Key-value pairs representing environment\n    variables. Values can be strings, numbers, or booleans.\n\n> [!TIP]\n>\n> If the user doesn't provide specific paths for scripts, dbt projects, or GCP\n> details (Project ID, Region), use tools like `find_by_name` to search the\n> repository and `gcloud` commands (e.g., `gcloud config get-value project`) to\n> retrieve the necessary information.\n\n### Step 3: Generate the pipeline files\n\n-   Before generating, check if an orchestration pipeline definition file and\n    `deployment.yaml` already exist in the current directory. If they do, inform\n    the user and ask if they want to update the existing files or create new\n    ones with different names. Do not overwrite without confirmation.\n\n-   First, before creating the orchestration pipeline definition file, you\n    **must** first run the following command to get the list of available\n    dataproc environments for the user's project. This avoids using placeholder\n    values to run the jobs.\n\n    ```\n    # Replace \u003CPROJECT_ID> with the actual project_id\n    # Replace \u003CREGION> with the actual region\n    gcloud dataproc clusters list \\\n    --project \u003CPROJECT_ID> \\\n    --region \u003CREGION> \\\n    ```\n\n    > [!TIP]\n    >\n    > Running the command without `--format=yaml` provides a clear, tabular\n    > output that is easier to read.\n\n-   Then use the returned dataproc list with details to create the orchestration\n    pipeline definition file based on the user's requirements for the pipeline's\n    logic and schedule. **IMPORTANT:** Every schedule **must** include an\n    `endTime`. Every schedule **must** use the current date as `startTime` if\n    the user hasn't specified.\n\n    > [!IMPORTANT]\n    >\n    > A Composer environment is not a Dataproc cluster. If no Dataproc clusters\n    > are available, do not use a Composer environment for the\n    > `sparkHistoryServerConfig`. It is better to omit this configuration if a\n    > dedicated Spark History Server is not available.\n\n-   If you want to schedule the python job, check the content of Python content\n    to determine if it's a spark job. If it is, use `pyspark` as type instead of\n    script as type.\n\n-   Before creating or updating the `deployment.yaml` file, you **must** first\n    run the following command to get the list of available Composer environments\n    for the user's project.\n\n    ```\n    # Replace \u003CPROJECT_ID> with the actual project_id\n    # Replace \u003CREGION> with the actual region\n    gcloud composer environments list \\\n    --project \u003CPROJECT_ID> \\\n    --locations \u003CREGION> \\\n    ```\n\n    After listing available Composer environments, you **must** check each\n    environment to ensure the composer is using the right image version or has\n    installed right PyPI packages. Run the following command for each\n    environment:\n\n    ```\n    # Replace \u003CENVIRONMENT_NAME> with the Composer environment name\n    # Replace \u003CREGION> with the region\n    gcloud composer environments describe \u003CENVIRONMENT_NAME> \\\n    --location \u003CREGION> \\\n    --format=\"json(config.softwareConfig.imageVersion, config.softwareConfig.pypiPackages)\"\n    ```\n\n    From the output, select an environment where the imageVersion value is one\n    of is \"composer-3-airflow-3.1.7-build.x, composer-3-airflow-2.11.1-build.x,\n    composer-3-airflow-2.10.5-build.x, composer-3-airflow-2.9.3-build.x,\n    composer-2.16.11-airflow-2.11.1, composer-2.16.11-airflow-2.10.5,\n    composer-2.16.11-airflow-2.9.3\" or select an environment\n    where`orchestration-pipelines` field is presented listed in the PyPI\n    packages. This ensures the selected environment is compatible with\n    orchestration pipelines.\n\n-   Third, before generating the `deployment.yaml` file, you **must ask the\n    user** to provide the `artifact_storage` bucket name. Note that the\n    `artifact_storage` bucket is typically initialized as a placeholder (e.g.,\n    `YOUR_BUCKET`) by the `init` command in Step 1. You must identify any such\n    placeholders, ask the user for the actual bucket name, and then update the\n    `deployment.yaml` file with the provided value.\n\n    Use the returned composer list with details, along with the project ID,\n    region, and the bucket name provided by the user, to generate or update the\n    `deployment.yaml` file. When generating or updating the `deployment.yaml`\n    file, you **must** replace placeholders (e.g., \"\u003CYOUR_PROJECT_ID>\",\n    \"\u003CYOUR_REGION>\", \"\u003CYOUR_COMPOSER>\", \"\u003CYOUR_BUCKET>\") with the actual\n    retrieved and provided values. Additionally, you **must** remove any\n    associated `# TODO:` comments once the placeholders are replaced.\n\n-   Ensure both files adhere to the code structures and syntax specified in this\n    document.\n\n-   **Renaming Pipelines**: If requested to change the orchestration pipeline\n    name, you must rename the orchestration YAML file accordingly (e.g., from\n    `dbt_clean_pipeline.yaml` to `new_name.yaml`) and update the `source` field\n    within the `pipelines` list in `deployment.yaml` to match the new filename.\n\n> [!IMPORTANT]\n>\n> **Time Format**: Do NOT include the `Z` suffix in `startTime` and `endTime`.\n> Use the format `\"YYYY-MM-DDTHH:MM:SS\"` (e.g., `\"2025-10-01T00:00:00\"`).\n\n### Step 4: Validate the content (REQUIRED)\n\nAfter creating or editing pipeline files, you **MUST** validate them using the\n`gcloud beta orchestration-pipelines validate` command. you must: a. Read the\n`deployment.yaml` file to identify all defined environments. b. Run the\n`validate` command below for **each** environment found in `deployment.yaml`.\n\n```\n# Replace \u003CENV_NAME> with the identified environment name\ngcloud beta orchestration-pipelines validate --environment=\u003CENV_NAME>\n```\n\n### Step 5: Handle Validation Errors\n\n1.  Check the output of the validation command.\n\n2.  If the command returns an error or failure message:\n\n    -   Read the error message carefully.\n    -   Edit the orchestration and deployment files to fix the specific issue\n        mentioned.\n\n3.  Re-run the validation command to confirm the fix. Do not mark the task as\n    complete until the validation passes (exit code 0), and do not fall back to\n    create airflow dag in python if validation fails.\n\n## Declarative Pipeline Templates\n\nWhen asked to generate or verify declarative pipeline files, ensure they follow\nthese compliant structures. **Do not use the exact values below; adapt them to\nthe user's specific project, region, and environment details.**\n\n### `deployment.yaml` Template - IMPORTANT FORMAT MUST MATCH-\n\n```yaml\nenvironments:\n  \u003Cenvironment_name>: # e.g., dev, prod\n    project: \u003CPROJECT_ID>\n    region: \u003CREGION>\n    composer_environment: \u003CCOMPOSER_ENVIRONMENT_NAME>\n    gcs_bucket: \"\" # Optional\n    artifact_storage:\n      bucket: \u003CARTIFACT_BUCKET_NAME>\n      path_prefix: \"\u003Cprefix>-\" # e.g., namespace or username prefix\n    pipelines:\n      - source: '\u003Corchestration-pipeline.yaml>' # e.g., list of pipeline yaml names\n```\n\n### Step 6: Deploy the Orchestration Pipeline (Optional)\n\nIf requested to **deploy** the orchestration pipeline:\n\n1.  You MUST ask the user which environment to deploy to. If no environment name\n    is provided, list the available environments from `deployment.yaml` and ask\n    the user to choose one, defaulting to `dev` if it exists.\n\n2.  Read the orchestration YAML to extract the `pipelineId`.\n\n3.  Deploy with `--local`. This uploads the DAG without running it:\n\n    ```\n    # Replace \u003CENV_NAME> with the target environment\n    # Replace \u003CPIPELINE_SOURCE> with the orchestration YAML filename\n    gcloud beta orchestration-pipelines deploy \\\n      --environment=\u003CENV_NAME> --local\n    ```\n\n4.  Parse the deploy output to extract the **bundle ID** (version). The output\n    includes a line like: `Pipeline deployment successful for version\n    local-b32d15e307b5` The version string (e.g., `local-b32d15e307b5`) is the\n    bundle ID.\n\n> [!IMPORTANT]\n>\n> `--local` deployments now default to `--paused=true`. The deployed DAG will be\n> visible in Airflow as a paused DAG without a schedule. It will **not**\n> auto-run. Use Step 7 to trigger it.\n\n### Step 7: Trigger the Orchestration Pipeline Run (Optional)\n\nIf requested to **trigger\u002Frun** the orchestration pipeline, you MUST follow the\nDeploy → Poll → Trigger flow.\n\n1.  **Ask for environment**: You MUST ask the user which environment to use.\n    Default to `dev` if it exists in `deployment.yaml`.\n\n2.  **Deploy first** (Step 6): Always deploy before triggering to ensure the run\n    uses the latest code. Extract the `bundle ID` from deploy output and the\n    `pipelineId` from the orchestration YAML.\n\n3.  **Poll for DAG readiness**: Wait for the DAG to be registered in Composer.\n\n    ```bash\n    # Initial delay: wait 30 seconds after deploy\n    sleep 30\n\n    # Poll every 15 seconds, up to 2 minutes total\n    # Replace \u003CENV_NAME>, \u003CBUNDLE_ID> with actual values\n\n    gcloud beta orchestration-pipelines list \\\n    --environment=\u003CENV_NAME> \\\n    --bundle=\u003CBUNDLE_ID>\n    ```\n\n    The pipeline is ready when it appears in the list output. If it does not\n    appear after 2 minutes, report failure and advise the user to check YAML\n    validity.\n\n4.  **Trigger the pipeline**:\n\n    ```\n    # Replace \u003CENV_NAME>, \u003CBUNDLE_ID>, \u003CPIPELINE_ID> with actual values\n    gcloud beta orchestration-pipelines trigger \\\n    --environment=\u003CENV_NAME> \\\n    --bundle=\u003CBUNDLE_ID> \\\n    --pipeline=\u003CPIPELINE_ID>\n    ```\n\n5.  **Verify the run started**:\n\n    ```\n    gcloud beta orchestration-pipelines runs list \\\n    --environment=\u003CENV_NAME> \\\n    --bundle=\u003CBUNDLE_ID> \\\n    --pipeline=\u003CPIPELINE_ID>\n    ```\n\n> [!TIP]\n>\n> **Trigger-only (no deploy):** If the user wants to trigger an already-deployed\n> pipeline, skip Step 6. Use `gcloud beta orchestration-pipelines list\n> --environment=\u003CENV_NAME>` to find the bundle ID, then trigger directly with\n> Step 7.4.\n\n> [!IMPORTANT]\n>\n> **Fallback:** If `gcloud trigger` fails, use the bundled script: Run script\n> with -- help to discover and learn the interface.\n>\n> ```\n> python scripts\u002Ftrigger\u002Fairflow_trigger.py \\ --project \u003CPROJECT_ID>\n> --location \u003CREGION> \\ --environment \u003CCOMPOSER_ENV> --dag_id \u003CPIPELINE_ID>\n> ```\n>\n> Get `project`, `region`, and `composer_environment` from `deployment.yaml`.\n\n## Definition of done\n\n-   `deployment.yaml` file is created successfully.\n-   The orchestration pipeline file (e.g., `orchestration_pipeline.yaml`) is\n    created successfully, includes a mandatory `endTime` for every schedule, and\n    passes the validation command: `gcloud beta orchestration-pipelines validate\n    --environment=\u003CENV_NAME>`\n-   If user requested to **deploy** the orchestration pipeline, the `gcloud beta\n    orchestration-pipelines deploy --environment=\u003CENV_NAME> --local` command\n    should return a success message with a version\u002Fbundle ID.\n-   If user requested to **trigger\u002Frun** the orchestration pipeline:\n    1.  Deploy succeeded (bundle ID extracted from output)\n    2.  DAG appeared in `gcloud beta orchestration-pipelines list` within 2 min\n    3.  `gcloud beta orchestration-pipelines trigger` returned success\n    4.  Run is visible in `gcloud beta orchestration-pipelines runs list`\n\n## Other actions\n\nIf requested to pause\u002Fstop the orchestration pipeline, use\n\n```bash\n    # Replace \u003CENV_NAME>, \u003CBUNDLE_ID>, \u003CPIPELINE_ID> with actual values\n    gcloud beta orchestration-pipelines pause \\\n    --environment=\u003CENV_NAME> \\\n    --bundle=\u003CBUNDLE_ID> \\\n    --pipeline=\u003CPIPELINE_ID>\n```\n\nIf requested to unpause\u002Fresume the orchestration pipeline, use\n\n```bash\n    # Replace \u003CENV_NAME>, \u003CBUNDLE_ID>, \u003CPIPELINE_ID> with actual values\n    gcloud beta orchestration-pipelines unpause \\\n    --environment=\u003CENV_NAME> \\\n    --bundle=\u003CBUNDLE_ID> \\\n    --pipeline=\u003CPIPELINE_ID>\n```\n","## Mandatory Reference Routing\n\nIf relevant, call the associated reference file(s) before you take actions.\nRefer to the table below to determine which reference file to retrieve in\ndifferent scenarios involving specific functions. [!IMPORTANT]: DO NOT GUESS\nfilenames. You MUST only use the exact paths provided below.\n\nFunction\u002FUse Case                  | Required Reference File                        | Capabilities & Intent Keywords\n---------------------------------- | ---------------------------------------------- | ------------------------------\n**orchestration-pipelines schema** | `references\u002Forchestration-pipelines-schema.md` | orchestrate, generate, create, update\n\n## How to use this skill\n\nOrchestration pipelines require creating two files to ensure a complete and\ndeployable pipeline:\n\n1.  `Orchestration File` (e.g., `orchestration-pipeline.yaml`,\n    `test-pipeline.yaml`): Defines the pipeline's logic, tasks, and schedule.\n    **IMPORTANT:** Check if a `deployment.yaml` file exists and references an\n    existing orchestration file. If it does, you **must update the existing\n    orchestration file** (e.g.,`test_pipeline.yaml`) instead of creating a new\n    one. The filename can be customized but must be referenced in the\n    `deployment.yaml` file.\n2.  `deployment.yaml`: Defines the environment-specific configurations.(e.g.,\n    `dev`, `prod`). `deployment.yaml`should only exists in the repository root\n    and must be named `deployment.yaml`\n\n-   All files should always be maintained together. And all files should be\n    placed on the root of the workspace folder.\n\n-   This skill is helpful to create or update configuration files to orchestrate\n    data pipelines.\n\n## How to use this skill\n\n### Step 1: Assess Orchestration Pipeline Status and Initialize if Necessary\n\nExamine the repository's root directory for a `deployment.yaml` file.\n\n1.  **Check for existing setup**: The absence of `deployment.yaml` indicates\n    that orchestration has not been set up.\n2.  **Determine if initialization is required**: Initialization is required if\n    `deployment.yaml` is missing. you **MUST** run the `init` command in Step 3\n    to scaffold the project if `deployment.yaml` is missing. Do NOT create the\n    files manually.\n3.  **Pipeline Name**: If initialization is needed, ask the user for the\n    pipeline name. If user hasn't provided the orchestration pipeline name, name\n    should be \"orchestration_pipeline\"\n4.  **Environment Name**: If initialization is needed, you MUST ask the user for\n    the environment name. If the user does not provide it, use **dev** as the\n    default.\n\n5.  **Execute Initialization**: Once you have the pipeline name, run the\n    following command:\n\n```\n# Replace \u003CORCHESTRATION_PIPELINE_NAME> with the actual name\n# Replace \u003CENV_NAME> with the actual environment name\ngcloud beta orchestration-pipelines init \u003CORCHESTRATION_PIPELINE_NAME> --environment=\u003CENV_NAME>\n```\n\n### Step 2: Review the orchestration pipeline code structure and syntax instruction\n\n*** Pipeline Models (mapping to YAML)\n\n> [!IMPORTANT]\n>\n> While the internal pipeline models are defined using protobuf (which typically\n> uses `snake_case`), the **YAML configuration expects `camelCase`** for almost\n> all field names.\n>\n> **Mapping Rule:** Always convert `snake_case` proto fields (e.g.,\n> `pipeline_id`) to `camelCase` in YAML (e.g., `pipelineId`).\n\n#### Orchestration-Pipelines yaml structure and syntax instruction\n\nReference to file `references\u002Forchestration-pipelines-schema.md`.\n\n**Required Tags (Top-Level)**: You **MUST** add a `tags` field to the top-level\nof the orchestration pipeline YAML definition. The value of this field depends\non the IDE environment:\n\n-   For Antigravity, use `[\"job:datacloud:antigravity\"]`.\n-   For VS Code, use `[\"job:datacloud:vscode\"]`.\n-   For Gemini CLI, use `[\"job:datacloud:gemini-cli\"]`.\n-   For Claude Code, use `[\"job:datacloud:claude\"]`.\n-   For Codex, use `[\"job:datacloud:codex\"]`.\n-   For any other environment, use `[\"job:datacloud:other\"]`.\n\n#### Deployment yaml structure and syntax instruction.\n\n**Top-Level Structure:** The root of the YAML should be an object with the\nfollowing fields:\n\n-   `environments` (dictionary): A map where keys are environment names (e.g.,\n    'dev', 'prod', etc) and values are Environment objects.\n\n**Environment:** Each environment object contains the following fields:\n\n-   `project` (string): The Google Cloud Project ID.\n-   `region` (string): The Google Cloud region (e.g., 'us-central1').\n-   `composer_environment` (string): The Cloud Composer environment name.\n-   `artifact_storage`\n    -   `bucket` (string): GCS bucket\n    -   `path_prefix`(string): prefix of path that we want to put in bucket\n-   `pipelines`\n    -   `- source` (string): orchestration pipeline yaml file names. It can be\n        multiple\n-   `variables` (dictionary, optional): Key-value pairs representing environment\n    variables. Values can be strings, numbers, or booleans.\n\n> [!TIP]\n>\n> If the user doesn't provide specific paths for scripts, dbt projects, or GCP\n> details (Project ID, Region), use tools like `find_by_name` to search the\n> repository and `gcloud` commands (e.g., `gcloud config get-value project`) to\n> retrieve the necessary information.\n\n### Step 3: Generate the pipeline files\n\n-   Before generating, check if an orchestration pipeline definition file and\n    `deployment.yaml` already exist in the current directory. If they do, inform\n    the user and ask if they want to update the existing files or create new\n    ones with different names. Do not overwrite without confirmation.\n\n-   First, before creating the orchestration pipeline definition file, you\n    **must** first run the following command to get the list of available\n    dataproc environments for the user's project. This avoids using placeholder\n    values to run the jobs.\n\n    ```\n    # Replace \u003CPROJECT_ID> with the actual project_id\n    # Replace \u003CREGION> with the actual region\n    gcloud dataproc clusters list \\\n    --project \u003CPROJECT_ID> \\\n    --region \u003CREGION> \\\n    ```\n\n    > [!TIP]\n    >\n    > Running the command without `--format=yaml` provides a clear, tabular\n    > output that is easier to read.\n\n-   Then use the returned dataproc list with details to create the orchestration\n    pipeline definition file based on the user's requirements for the pipeline's\n    logic and schedule. **IMPORTANT:** Every schedule **must** include an\n    `endTime`. Every schedule **must** use the current date as `startTime` if\n    the user hasn't specified.\n\n    > [!IMPORTANT]\n    >\n    > A Composer environment is not a Dataproc cluster. If no Dataproc clusters\n    > are available, do not use a Composer environment for the\n    > `sparkHistoryServerConfig`. It is better to omit this configuration if a\n    > dedicated Spark History Server is not available.\n\n-   If you want to schedule the python job, check the content of Python content\n    to determine if it's a spark job. If it is, use `pyspark` as type instead of\n    script as type.\n\n-   Before creating or updating the `deployment.yaml` file, you **must** first\n    run the following command to get the list of available Composer environments\n    for the user's project.\n\n    ```\n    # Replace \u003CPROJECT_ID> with the actual project_id\n    # Replace \u003CREGION> with the actual region\n    gcloud composer environments list \\\n    --project \u003CPROJECT_ID> \\\n    --locations \u003CREGION> \\\n    ```\n\n    After listing available Composer environments, you **must** check each\n    environment to ensure the composer is using the right image version or has\n    installed right PyPI packages. Run the following command for each\n    environment:\n\n    ```\n    # Replace \u003CENVIRONMENT_NAME> with the Composer environment name\n    # Replace \u003CREGION> with the region\n    gcloud composer environments describe \u003CENVIRONMENT_NAME> \\\n    --location \u003CREGION> \\\n    --format=\"json(config.softwareConfig.imageVersion, config.softwareConfig.pypiPackages)\"\n    ```\n\n    From the output, select an environment where the imageVersion value is one\n    of is \"composer-3-airflow-3.1.7-build.x, composer-3-airflow-2.11.1-build.x,\n    composer-3-airflow-2.10.5-build.x, composer-3-airflow-2.9.3-build.x,\n    composer-2.16.11-airflow-2.11.1, composer-2.16.11-airflow-2.10.5,\n    composer-2.16.11-airflow-2.9.3\" or select an environment\n    where`orchestration-pipelines` field is presented listed in the PyPI\n    packages. This ensures the selected environment is compatible with\n    orchestration pipelines.\n\n-   Third, before generating the `deployment.yaml` file, you **must ask the\n    user** to provide the `artifact_storage` bucket name. Note that the\n    `artifact_storage` bucket is typically initialized as a placeholder (e.g.,\n    `YOUR_BUCKET`) by the `init` command in Step 1. You must identify any such\n    placeholders, ask the user for the actual bucket name, and then update the\n    `deployment.yaml` file with the provided value.\n\n    Use the returned composer list with details, along with the project ID,\n    region, and the bucket name provided by the user, to generate or update the\n    `deployment.yaml` file. When generating or updating the `deployment.yaml`\n    file, you **must** replace placeholders (e.g., \"\u003CYOUR_PROJECT_ID>\",\n    \"\u003CYOUR_REGION>\", \"\u003CYOUR_COMPOSER>\", \"\u003CYOUR_BUCKET>\") with the actual\n    retrieved and provided values. Additionally, you **must** remove any\n    associated `# TODO:` comments once the placeholders are replaced.\n\n-   Ensure both files adhere to the code structures and syntax specified in this\n    document.\n\n-   **Renaming Pipelines**: If requested to change the orchestration pipeline\n    name, you must rename the orchestration YAML file accordingly (e.g., from\n    `dbt_clean_pipeline.yaml` to `new_name.yaml`) and update the `source` field\n    within the `pipelines` list in `deployment.yaml` to match the new filename.\n\n> [!IMPORTANT]\n>\n> **Time Format**: Do NOT include the `Z` suffix in `startTime` and `endTime`.\n> Use the format `\"YYYY-MM-DDTHH:MM:SS\"` (e.g., `\"2025-10-01T00:00:00\"`).\n\n### Step 4: Validate the content (REQUIRED)\n\nAfter creating or editing pipeline files, you **MUST** validate them using the\n`gcloud beta orchestration-pipelines validate` command. you must: a. Read the\n`deployment.yaml` file to identify all defined environments. b. Run the\n`validate` command below for **each** environment found in `deployment.yaml`.\n\n```\n# Replace \u003CENV_NAME> with the identified environment name\ngcloud beta orchestration-pipelines validate --environment=\u003CENV_NAME>\n```\n\n### Step 5: Handle Validation Errors\n\n1.  Check the output of the validation command.\n\n2.  If the command returns an error or failure message:\n\n    -   Read the error message carefully.\n    -   Edit the orchestration and deployment files to fix the specific issue\n        mentioned.\n\n3.  Re-run the validation command to confirm the fix. Do not mark the task as\n    complete until the validation passes (exit code 0), and do not fall back to\n    create airflow dag in python if validation fails.\n\n## Declarative Pipeline Templates\n\nWhen asked to generate or verify declarative pipeline files, ensure they follow\nthese compliant structures. **Do not use the exact values below; adapt them to\nthe user's specific project, region, and environment details.**\n\n### `deployment.yaml` Template - IMPORTANT FORMAT MUST MATCH-\n\n```yaml\nenvironments:\n  \u003Cenvironment_name>: # e.g., dev, prod\n    project: \u003CPROJECT_ID>\n    region: \u003CREGION>\n    composer_environment: \u003CCOMPOSER_ENVIRONMENT_NAME>\n    gcs_bucket: \"\" # Optional\n    artifact_storage:\n      bucket: \u003CARTIFACT_BUCKET_NAME>\n      path_prefix: \"\u003Cprefix>-\" # e.g., namespace or username prefix\n    pipelines:\n      - source: '\u003Corchestration-pipeline.yaml>' # e.g., list of pipeline yaml names\n```\n\n### Step 6: Deploy the Orchestration Pipeline (Optional)\n\nIf requested to **deploy** the orchestration pipeline:\n\n1.  You MUST ask the user which environment to deploy to. If no environment name\n    is provided, list the available environments from `deployment.yaml` and ask\n    the user to choose one, defaulting to `dev` if it exists.\n\n2.  Read the orchestration YAML to extract the `pipelineId`.\n\n3.  Deploy with `--local`. This uploads the DAG without running it:\n\n    ```\n    # Replace \u003CENV_NAME> with the target environment\n    # Replace \u003CPIPELINE_SOURCE> with the orchestration YAML filename\n    gcloud beta orchestration-pipelines deploy \\\n      --environment=\u003CENV_NAME> --local\n    ```\n\n4.  Parse the deploy output to extract the **bundle ID** (version). The output\n    includes a line like: `Pipeline deployment successful for version\n    local-b32d15e307b5` The version string (e.g., `local-b32d15e307b5`) is the\n    bundle ID.\n\n> [!IMPORTANT]\n>\n> `--local` deployments now default to `--paused=true`. The deployed DAG will be\n> visible in Airflow as a paused DAG without a schedule. It will **not**\n> auto-run. Use Step 7 to trigger it.\n\n### Step 7: Trigger the Orchestration Pipeline Run (Optional)\n\nIf requested to **trigger\u002Frun** the orchestration pipeline, you MUST follow the\nDeploy → Poll → Trigger flow.\n\n1.  **Ask for environment**: You MUST ask the user which environment to use.\n    Default to `dev` if it exists in `deployment.yaml`.\n\n2.  **Deploy first** (Step 6): Always deploy before triggering to ensure the run\n    uses the latest code. Extract the `bundle ID` from deploy output and the\n    `pipelineId` from the orchestration YAML.\n\n3.  **Poll for DAG readiness**: Wait for the DAG to be registered in Composer.\n\n    ```bash\n    # Initial delay: wait 30 seconds after deploy\n    sleep 30\n\n    # Poll every 15 seconds, up to 2 minutes total\n    # Replace \u003CENV_NAME>, \u003CBUNDLE_ID> with actual values\n\n    gcloud beta orchestration-pipelines list \\\n    --environment=\u003CENV_NAME> \\\n    --bundle=\u003CBUNDLE_ID>\n    ```\n\n    The pipeline is ready when it appears in the list output. If it does not\n    appear after 2 minutes, report failure and advise the user to check YAML\n    validity.\n\n4.  **Trigger the pipeline**:\n\n    ```\n    # Replace \u003CENV_NAME>, \u003CBUNDLE_ID>, \u003CPIPELINE_ID> with actual values\n    gcloud beta orchestration-pipelines trigger \\\n    --environment=\u003CENV_NAME> \\\n    --bundle=\u003CBUNDLE_ID> \\\n    --pipeline=\u003CPIPELINE_ID>\n    ```\n\n5.  **Verify the run started**:\n\n    ```\n    gcloud beta orchestration-pipelines runs list \\\n    --environment=\u003CENV_NAME> \\\n    --bundle=\u003CBUNDLE_ID> \\\n    --pipeline=\u003CPIPELINE_ID>\n    ```\n\n> [!TIP]\n>\n> **Trigger-only (no deploy):** If the user wants to trigger an already-deployed\n> pipeline, skip Step 6. Use `gcloud beta orchestration-pipelines list\n> --environment=\u003CENV_NAME>` to find the bundle ID, then trigger directly with\n> Step 7.4.\n\n> [!IMPORTANT]\n>\n> **Fallback:** If `gcloud trigger` fails, use the bundled script: Run script\n> with -- help to discover and learn the interface.\n>\n> ```\n> python scripts\u002Ftrigger\u002Fairflow_trigger.py \\ --project \u003CPROJECT_ID>\n> --location \u003CREGION> \\ --environment \u003CCOMPOSER_ENV> --dag_id \u003CPIPELINE_ID>\n> ```\n>\n> Get `project`, `region`, and `composer_environment` from `deployment.yaml`.\n\n## Definition of done\n\n-   `deployment.yaml` file is created successfully.\n-   The orchestration pipeline file (e.g., `orchestration_pipeline.yaml`) is\n    created successfully, includes a mandatory `endTime` for every schedule, and\n    passes the validation command: `gcloud beta orchestration-pipelines validate\n    --environment=\u003CENV_NAME>`\n-   If user requested to **deploy** the orchestration pipeline, the `gcloud beta\n    orchestration-pipelines deploy --environment=\u003CENV_NAME> --local` command\n    should return a success message with a version\u002Fbundle ID.\n-   If user requested to **trigger\u002Frun** the orchestration pipeline:\n    1.  Deploy succeeded (bundle ID extracted from output)\n    2.  DAG appeared in `gcloud beta orchestration-pipelines list` within 2 min\n    3.  `gcloud beta orchestration-pipelines trigger` returned success\n    4.  Run is visible in `gcloud beta orchestration-pipelines runs list`\n\n## Other actions\n\nIf requested to pause\u002Fstop the orchestration pipeline, use\n\n```bash\n    # Replace \u003CENV_NAME>, \u003CBUNDLE_ID>, \u003CPIPELINE_ID> with actual values\n    gcloud beta orchestration-pipelines pause \\\n    --environment=\u003CENV_NAME> \\\n    --bundle=\u003CBUNDLE_ID> \\\n    --pipeline=\u003CPIPELINE_ID>\n```\n\nIf requested to unpause\u002Fresume the orchestration pipeline, use\n\n```bash\n    # Replace \u003CENV_NAME>, \u003CBUNDLE_ID>, \u003CPIPELINE_ID> with actual values\n    gcloud beta orchestration-pipelines unpause \\\n    --environment=\u003CENV_NAME> \\\n    --bundle=\u003CBUNDLE_ID> \\\n    --pipeline=\u003CPIPELINE_ID>\n```\n",{"name":5,"license":14,"metadata":23,"description":7},{"version":24,"publisher":25},"v1","google","main","900ea3c9eb045fd5b0317dc1103777ecf3ea4aeb","skills\u002Fgcp_pipeline_orchestration",0,"https:\u002F\u002Fgithub.com\u002Fnguyenhungtran18\u002Fskill-and-tool-tracker","https:\u002F\u002Fnguyenhungtran18.github.io\u002Fskill-and-tool-tracker\u002F","https:\u002F\u002Fgithub.com\u002Fnguyenhungtran18","2026-08-31 18:36:09",{"source":8,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add nguyenhungtran18\u002Fskill-and-tool-tracker --skill gcp-pipeline-orchestration",{"tool":40,"command":41},"bunx","bunx skills add nguyenhungtran18\u002Fskill-and-tool-tracker --skill gcp-pipeline-orchestration",{"tool":43,"command":44},"pnpm","pnpm dlx skills add nguyenhungtran18\u002Fskill-and-tool-tracker --skill gcp-pipeline-orchestration","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fnguyenhungtran18\u002Fskill-and-tool-tracker\u002Fblob\u002Fmain\u002Fskills\u002Fgcp_pipeline_orchestration\u002FSKILL.md",true,[48,54],{"path":49,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":52,"sourceHtmlUrl":53},"references\u002Forchestration-pipelines-schema.md","REFERENCE","参考文档",14445,"https:\u002F\u002Fgithub.com\u002Fnguyenhungtran18\u002Fskill-and-tool-tracker\u002Fblob\u002Fmain\u002Fskills\u002Fgcp_pipeline_orchestration\u002Freferences\u002Forchestration-pipelines-schema.md",{"path":55,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":58,"sourceHtmlUrl":59},"scripts\u002Ftrigger\u002Fairflow_trigger.py","SCRIPT","脚本",4582,"https:\u002F\u002Fgithub.com\u002Fnguyenhungtran18\u002Fskill-and-tool-tracker\u002Fblob\u002Fmain\u002Fskills\u002Fgcp_pipeline_orchestration\u002Fscripts\u002Ftrigger\u002Fairflow_trigger.py",[61,69,76,83,90,97],{"skillNo":62,"name":63,"nameZh":6,"description":64,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":65,"licenseFileUrl":6,"sourceHtmlUrl":66,"categories":67,"keywords":68,"publishedAt":18},"01M7BVP4P83TK60JZ4P8RP9VP7","xcode-project-setup","Safely modifies Xcode projects (.pbxproj) to add Swift Packages and link files. Use this skill whenever an iOS project needs dependencies installed (e.g. Firebase, Alamofire).","MIT","https:\u002F\u002Fgithub.com\u002Fnguyenhungtran18\u002Fskill-and-tool-tracker\u002Fblob\u002Fmain\u002Fskills\u002Fxcode_project_setup\u002FSKILL.md",[],[],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":65,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":18},"01M7BVP01K4CSJJ8G90FT8S2D5","writing-skills","Use when creating new skills, editing existing skills, or verifying skills work before deployment","https:\u002F\u002Fgithub.com\u002Fnguyenhungtran18\u002Fskill-and-tool-tracker\u002Fblob\u002Fmain\u002Fskills\u002Fwriting-skills\u002FSKILL.md",[],[],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":65,"licenseFileUrl":6,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":18},"01M7BVNY22GXD871GYTSH599F1","writing-plans","Use when you have a spec or requirements for a multi-step task, before touching code","https:\u002F\u002Fgithub.com\u002Fnguyenhungtran18\u002Fskill-and-tool-tracker\u002Fblob\u002Fmain\u002Fskills\u002Fwriting-plans\u002FSKILL.md",[],[],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":65,"licenseFileUrl":6,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":18},"01M7BVNQWF24RNKA8A4KM2HJ3T","workflow-skill-creator","Distills a completed user workflow or interaction into a reusable agent skill. Use when the user asks to turn their workflow, interaction, or multi-step process into a skill, or when they say \"make this a skill\", \"create a skill from what we just did\", \"package this workflow\" or similar. Do not use for creating skills from scratch without an existing workflow (use a generic skill-creator for that).","https:\u002F\u002Fgithub.com\u002Fnguyenhungtran18\u002Fskill-and-tool-tracker\u002Fblob\u002Fmain\u002Fskills\u002Fworkflow_skill_creator\u002FSKILL.md",[],[],{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":65,"licenseFileUrl":6,"sourceHtmlUrl":94,"categories":95,"keywords":96,"publishedAt":18},"01M7BVNNY67XPCQAQHY3FBKDGC","verify-and-stop","Prove existing work meets acceptance conditions without expanding scope. Use for validation-only tasks, completion checks, focused gate runs, and last-mile proof.","https:\u002F\u002Fgithub.com\u002Fnguyenhungtran18\u002Fskill-and-tool-tracker\u002Fblob\u002Fmain\u002Fskills\u002Fverify-and-stop\u002FSKILL.md",[],[],{"skillNo":98,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":65,"licenseFileUrl":6,"sourceHtmlUrl":101,"categories":102,"keywords":103,"publishedAt":18},"01M7BVNKB4Z7SZT85B73ZZ90Y6","verification-before-completion","Use when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always","https:\u002F\u002Fgithub.com\u002Fnguyenhungtran18\u002Fskill-and-tool-tracker\u002Fblob\u002Fmain\u002Fskills\u002Fverification-before-completion\u002FSKILL.md",[],[]]