[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6C38JEVFSSKA5E5KWTCSA6F":3,"skill-same:01M6C38JEVFSSKA5E5KWTCSA6F":62},{"skillNo":4,"name":5,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":6,"repositoryNo":11,"starsCount":12,"licenseSpdxId":13,"licenseFileUrl":6,"sourceHtmlUrl":14,"categories":15,"keywords":16,"publishedAt":24,"linkOnly":25,"markdownContent":26,"markdownBody":27,"frontMatter":28,"refName":29,"commitSha":30,"skillDirectory":31,"forksCount":32,"repositoryHtmlUrl":33,"repositoryHomepageUrl":6,"ownerHtmlUrl":34,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":35,"install":36,"resources":49},"01M6C38JEVFSSKA5E5KWTCSA6F","gcp-cost-diagnosis-bq-billing-export",null,"jleechanorg\u002Fjleechan-skills","jleechanorg","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F221404966?v=4","01M6C1VJ99XG950X1DWPEBCBGX","01M6C1VJDK5YVDNT3362RT5WSB",6,"MIT","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002Fhermes\u002Fskills\u002Fdevops\u002Fgcp-cost-diagnosis-bq-billing-export\u002FSKILL.md",[],[17,18,19,20,21,22,23],"gcp","export","gcp cost diagnosis bq billing export","cost","bq","diagnosis","billing","2026-11-10 10:12:30",false,"---\nname: gcp-cost-diagnosis-bq-billing-export\ndescription: Diagnose GCP daily cost spikes from the BQ billing export when the user asks \"why is X service so expensive\", \"why did GCP cost go up\", \"drill into Cloud Run spend\", or any per-service or per-SKU cost breakdown for a GCP project. Encodes the working pattern when `bq query` returns the misleading \"Aggregations of aggregations are not allowed\" error on STRUCT field aggregations - query per-line (no GROUP BY) via subprocess, aggregate in Python. Also covers the cost-floor-vs-variable distinction (Cloud Run min-instance memory vs request memory, etc.) which is the fastest way to tell whether a daily spike is a new instance-footprint problem or just a traffic spike. Verified 2026-07-17 against project `worldarchitecture-ai` billing_export.gcp_billing_export_v1_011269_D08BDB_79D8F2 (Cloud Run $14.28\u002Fday breakdown with $8.93 min-inst-mem + $3.20 min-inst-CPU = $12.13 idle floor vs $2.09 request-driven).\nversion: 1.0.0\nauthor: Hermes Agent\nlicense: MIT\nplatforms: [macos, linux]\nmetadata:\n  hermes:\n    tags: [gcp, billing, bigquery, cost-optimization, cloud-run, devops]\n    related_skills: [gh-actions-slow-runs, spend-alert-bridge, wa-cloud-run-deploy-failure-debug, mac-di[REDACTED_OPENAI_KEY]]\n---\n\n# GCP cost diagnosis from BQ billing export\n\nWhen the user asks \"why is GCP X expensive\" or \"drill into \u003Cservice> cost\" and the answer needs per-SKU granularity (not just the Cloud Console Subtotal), the BQ billing export is the source of truth. This skill encodes the **two non-obvious failure modes** that block the obvious approach and the working pattern that gets around them.\n\n## When this skill fires\n\n- \"Why is Cloud Run so expensive\" \u002F \"why did GCP cost spike\" \u002F \"drill into \u003Cservice> cost\"\n- The user shares a `[Daily GCP cost] YYYY-MM-DD — $X.XX` email (the worldarchitect daily cron template) and wants service breakdown\n- The user pastes a GCP cost table screenshot and asks about a specific service line\n- A spend-alert cron fires (see `spend-alert-bridge`) and the named mechanism points at a specific GCP service (Cloud Run, Cloud SQL, Cloud Storage, etc.)\n- Any question that requires per-SKU or per-service breakdown of GCP spend over a specific day or trend window\n\n**Anti-trigger:** if the user wants the **official billed figure** (for invoicing \u002F accounting), use the Cloud Console Subtotal directly. The BQ export runs **10-30% higher** than the Subtotal for ~2-3 weeks after a day closes because GCP keeps re-loading late-arriving usage events. State this caveat once when reporting BQ numbers.\n\n## The two failure modes\n\n### Failure mode 1 - `bq query` returns misleading \"Aggregations of aggregations are not allowed\"\n\nThe billing export schema has nested STRUCT fields (`service.description`, `sku.description`, `project.name`, `usage.amount`, `usage.unit`). When you write what looks like a normal GROUP BY query against this table:\n\n```sql\nSELECT sku.description AS sku, SUM(cost) AS cost\nFROM `project.billing_export.gcp_billing_export_v1_*`\nWHERE service.description = 'Cloud Run'\nGROUP BY sku\n```\n\n`bq query` returns:\n\n```\nError in query string: Error processing job ...: Aggregations of aggregations are not allowed at [9:12]\n```\n\nThe error message points at `SUM(cost)` but the query is syntactically valid (verified by running the SAME query via the python `google-cloud-bigquery` client - it works fine). The CLI parser hits some edge case with the heredoc \u002F STRUCT field combination.\n\n**The actual problem is not aggregations.** The query is fine. The CLI is misreporting.\n\n**The workaround that always works:**\n\n1. Drop the `GROUP BY` - select per-line items only.\n2. Pipe through `bq query --format=json`.\n3. Aggregate in Python (`collections.defaultdict` is fine - no `pandas` needed for thousands of rows).\n\n**The query that works (no GROUP BY):**\n\n```sql\nSELECT\n  sku.description AS sku,\n  usage.unit AS unit,\n  usage.pricing_unit AS pricing_unit,\n  cost,\n  usage.amount AS usage_qty\nFROM `PROJECT.billing_export.gcp_billing_export_v1_BILLINGACCT_HASH`\nWHERE usage_start_time >= TIMESTAMP('YYYY-MM-DD 00:00:00 UTC')\n  AND usage_start_time \u003C  TIMESTAMP('YYYY-MM-DD+1 00:00:00 UTC')\n  AND service.description = 'SERVICE_NAME'\nORDER BY cost DESC\nLIMIT 2000\n```\n\nThen aggregate in Python:\n\n```python\nimport subprocess, json\nfrom collections import defaultdict\n\nr = subprocess.run(\n    ['bq', 'query', '--use_legacy_sql=false', '--format=json', '--max_rows=2000'],\n    input=open('\u002Ftmp\u002Fq.sql').read(), capture_output=True, text=True, timeout=60,\n)\ndata = json.loads(r.stdout)\n\nagg = defaultdict(lambda: {'cost': 0.0, 'usage': 0.0, 'unit': '', 'lines': 0})\nfor row in data:\n    agg[row['sku']]['cost'] += float(row['cost'])\n    agg[row['sku']]['usage'] += float(row['usage_qty'])\n    agg[row['sku']]['unit'] = row.get('unit') or row.get('pricing_unit') or ''\n    agg[row['sku']]['lines'] += 1\n\ntotal = sum(v['cost'] for v in agg.values())\nfor sku, v in sorted(agg.items(), key=lambda x: -x[1]['cost']):\n    print(f\"{sku[:69]:\u003C70} {str(v['unit'])[:14]:\u003C15} ${v['cost']:>8.2f}\")\nprint(f\"TOTAL ${total:.2f}\")\n```\n\n### Failure mode 2 - `bq` subprocess can't find its `utils` module when launched from a sandbox\n\nIf you call `subprocess.run(['bq', 'query', ...])` from inside an `execute_code` sandbox without sourcing `~\u002F.bashrc`, the gcloud SDK's `utils` import fails with:\n\n```\nImportError: cannot import name 'bq_error' from 'utils'\n```\n\n**Fix:** source `~\u002F.bashrc` first (which sets up the gcloud SDK env), or run `bq query` from a `terminal()` tool call (which uses the user's normal shell).\n\n```bash\nbash -c 'source ~\u002F.bashrc 2>\u002Fdev\u002Fnull; bq query --use_legacy_sql=false --format=json \u003C \u002Ftmp\u002Fq.sql > \u002Ftmp\u002Fq.json 2>\u002Ftmp\u002Fq.err; echo exit=$?'\n```\n\nThe `terminal()` tool with bash wrapping is the reliable path.\n\n## The discovery pattern - find the table + project + billing account\n\nBefore any cost query, you need three identifiers. Get them in this order:\n\n```bash\n# 1. Project (default if not multi-project)\ngcloud config get-value project\n\n# 2. Billing dataset name (NOT always called \"billing\")\nbq ls -d --project_id=PROJECT\n# Look for: billing_export \u002F billing \u002F cloud_billing_export\n\n# 3. Full billing export table name (note the long hash suffix)\nbq ls DATASET\n# Pick the table starting with `gcp_billing_export_v1_` (not `_resource_v1_`)\n\n# 4. Confirm schema (column names are STRUCT - see Failure mode 1)\nbq show DATASET.gcp_billing_export_v1_HASH | head -50\n```\n\n**Critical detail:** The billing export table name has a long hex suffix (`gcp_billing_export_v1_011269_D08BDB_79D8F2`). It's tied to the **billing account**, not the project. If you support multiple billing accounts (e.g. dev vs prod), each one has its own table. Don't try to wildcard with `gcp_billing_export_v1_*` - the wildcard works in some clients and fails in `bq query` heredocs.\n\n## The diagnosis framework - cost-floor vs cost-variable\n\nOnce you have the per-SKU breakdown for a single day, classify each SKU into one of two buckets. This tells you whether the cost is structural (idle fleet) or proportional (traffic):\n\n| Bucket | Indicator SKUs | Diagnosis direction |\n|---|---|---|\n| **Floor** (idle baseline) | \"Min Instance Memory\", \"Min Instance CPU\", \"Idle VM time\", \"Always Free tier overage\" | These scale with **fleet size x uptime**, not traffic. If floor > 70% of service spend, the fix is **right-size \u002F scale-to-zero \u002F stop running excess services** |\n| **Variable** (per-request) | \"CPU (Request-based billing)\", \"Memory (Request-based billing)\", \"Requests\", \"Network egress\" | These scale with traffic. Diagnose with traffic dashboards \u002F recent deploy spike |\n\n**Rule of thumb:** if floor SKUs dominate, the cost is **structural** and the fix is at the service-config layer (Cloud Run min-instance, VM rightsizing, etc.). If variable SKUs dominate, the cost is **proportional** and the fix is at the application \u002F traffic layer.\n\n**Verified example (Cloud Run, project `worldarchitecture-ai`, 2026-07-16):**\n\n| SKU | Cost | Bucket |\n|---|---:|---|\n| Services Min Instance Memory | $8.93 | **floor** |\n| Services Min Instance CPU | $3.20 | **floor** |\n| Services CPU (Request-based) | $1.57 | variable |\n| Services Memory (Request-based) | $0.52 | variable |\n| Network egress + Jobs + Requests | $0.06 | variable |\n| **Total** | **$14.28** | **floor = $12.13 (85%)** |\n\nFloor dominance means the user should look at fleet right-sizing, not traffic. For this fleet, `mvp-site-app-s1` through `s10` (10 staging slots) plus `dev`\u002F`preview`\u002F`stable`\u002F`staging` (4 slots) at 4 vCPU \u002F 16 GiB each = ~14 large slots held warm. Downsizing dev\u002Fpreview\u002Fstaging would drop the floor by ~50%.\n\n## The trend query - confirm \"is this new or chronic?\"\n\nIf the user asks \"why did X spike today\" but actually the same floor has been there for weeks, you need a 7-day trend. Same query shape, but with `usage_start_time >= TIMESTAMP(YYYY-MM-DD-N 00:00:00 UTC)` for a window:\n\n```sql\nSELECT\n  DATE(usage_start_time) AS day,\n  ROUND(SUM(CASE WHEN sku.description LIKE 'Services Min Instance Memory%' THEN cost ELSE 0 END), 2) AS min_inst_mem,\n  ROUND(SUM(CASE WHEN sku.description LIKE 'Services Min Instance CPU%' THEN cost ELSE 0 END), 2) AS min_inst_cpu,\n  ROUND(SUM(CASE WHEN sku.description LIKE 'Services CPU (Request%' THEN cost ELSE 0 END), 2) AS req_cpu,\n  ROUND(SUM(CASE WHEN sku.description LIKE 'Services Memory (Request%' THEN cost ELSE 0 END), 2) AS req_mem,\n  ROUND(SUM(cost), 2) AS total\nFROM `PROJECT.billing_export.gcp_billing_export_v1_HASH`\nWHERE service.description = 'SERVICE_NAME'\n  AND usage_start_time >= TIMESTAMP('YYYY-MM-DD-7 00:00:00 UTC')\n  AND usage_start_time \u003C  TIMESTAMP('YYYY-MM-DD+1 00:00:00 UTC')\nGROUP BY day\nORDER BY day\n```\n\nIf the trend shows the **floor** is the same every day and **variable** is what spiked, the user has a traffic anomaly to investigate. If the floor itself is rising day-over-day, they have a fleet-size drift (new services coming online, autoscaler floor creeping up, etc.).\n\n## Per-service attribution - the labels gap\n\n**Heads up:** Cloud Run billing export rows do NOT carry a per-service label by default. The export has `project.labels`, `labels`, and `system_labels` arrays, but they're typically empty unless someone manually tagged each Cloud Run service with `gcloud run services update SERVICE --update-labels=service_name=SERVICE`.\n\nIf labels are present, you can attribute cost per service via:\n\n```sql\nSELECT\n  -- labels is REPEATED RECORD with key\u002Fvalue - extract via UNNEST\n  l.value AS service_label,\n  SUM(cost) AS cost\nFROM `PROJECT.billing_export.gcp_billing_export_v1_HASH`,\n     UNNEST(labels) AS l\nWHERE l.key = 'service_name'\n  AND service.description = 'Cloud Run'\n  AND usage_start_time >= ...\nGROUP BY service_label\nORDER BY cost DESC\n```\n\nIf labels are NOT present (the common case), the workaround is to **cross-reference with `gcloud run services list`** and reason about which services are running with what min-instance config, then map back to the SKU breakdown by memory\u002FCPU footprint. This is what we did in the verified example - found 14 services at 4 vCPU\u002F16 GiB by running `gcloud run services list --format=table(...)` then explained the $8.93 min-instance-memory line as that fleet's idle footprint.\n\n**Recipe for the gcloud inventory:**\n\n```bash\ngcloud run services list --project=PROJECT --region=REGION \\\n  --format=\"table(metadata.name,metadata.annotations['autoscaling.knative.dev\u002FminScale'],spec.template.spec.containers[0].resources.limits)\"\n```\n\nAdd a `MIN_SCALE` annotation column if you want to see which services have min-instance set (vs default 0 = scale-to-zero).\n\n## Pitfalls\n\n### Pitfall 1: Trusting `bq query` error messages at face value\n\nThe \"Aggregations of aggregations are not allowed\" error is a red herring. The query is fine. The CLI parser is misreporting. Always cross-check by running the same query through the Python client or by stripping the GROUP BY. If the no-GROUP-BY version returns rows and the GROUP BY version errors with that exact message, you're hitting the CLI bug - not a real syntax error.\n\n### Pitfall 2: Forgetting the late-arrival caveat\n\nThe BQ number runs **10-30% higher** than the Cloud Console Subtotal for 2-3 weeks after a day closes. When reporting the BQ number, state this caveat once. If the user is comparing against a Cloud Console screenshot, explain that:\n- BQ = forward-looking, accumulates late events\n- Cloud Console Subtotal = point-in-time, more authoritative for \"what got billed today\"\n- Both will converge once the invoice closes\n\n### Pitfall 3: Aggregating on `usage.unit` directly\n\n`usage.unit` (e.g. \"byte-seconds\") is the raw measurement unit. `usage.pricing_unit` (e.g. \"gibibyte\") is the billing unit. When computing per-unit cost, use `usage.amount_in_pricing_units` (the value scaled to pricing_unit), NOT `usage.amount` (raw). Otherwise you get nonsense rates.\n\n### Pitfall 4: Treating all Cloud Run spend as \"traffic-driven\"\n\nCloud Run's two SKU families look similar in name (\"Services Memory\" vs \"Services Min Instance Memory\") but bill completely differently. If you only see \"Cloud Run is $14\u002Fday\" without breaking down by SKU, you'll misdiagnose every cost-spike report. Always pull the per-SKU table before forming a hypothesis.\n\n### Pitfall 5: Running `bq query` from a subprocess without sourcing bashrc\n\nThe gcloud SDK's `bq` command depends on a `utils` import that breaks when the env doesn't have the SDK path. Symptom: `ImportError: cannot import name 'bq_error' from 'utils'`. Always wrap in `bash -c 'source ~\u002F.bashrc 2>\u002Fdev\u002Fnull; bq query ...'`.\n\n### Pitfall 6: Missing the cost-floor vs variable distinction when reporting\n\nIf you only report the per-SKU table without naming which SKUs are floor vs variable, the user has to do the analysis themselves. Always group SKUs into the two buckets and state the floor-share ratio. \"85% floor \u002F 15% variable\" is the single most useful sentence in any Cloud Run cost-diagnosis reply.\n\n## Reference and cross-links\n\n- `references\u002F2026-07-17-cloud-run-cost-breakdown.md` - session-specific evidence: full per-SKU table for `worldarchitecture-ai` Cloud Run on 2026-07-16, gcloud service inventory (14 services at 4 vCPU\u002F16 GiB), 7-day trend showing floor dominance, and the resulting recommended fix (downsize dev\u002Fpreview\u002Fs1-s10\u002Fstaging).\n- `~\u002F.hermes\u002Fskills\u002Fgh-actions-slow-runs\u002FSKILL.md` - sibling skill for GH Actions spend spikes (different export, different scope).\n- `~\u002F.hermes\u002Fskills\u002Fspend-alert-bridge\u002FSKILL.md` - verify-alert discipline; use this skill AFTER spend-alert-bridge confirms the alert is real.\n- `~\u002F.hermes\u002Fskills\u002Fworldarchitect\u002Fwa-cloud-run-deploy-failure-debug\u002FSKILL.md` - Cloud Run *deploy* failure diagnosis (NOT cost). Different concern, same project.\n- `~\u002F.hermes\u002Fskills\u002Fmac-di[REDACTED_OPENAI_KEY]\u002FSKILL.md` - the di[REDACTED_OPENAI_KEY] sibling skill (different resource, same \"first-use three concurrent lanes\" pattern).\n\n## One-line summary\n\n**Query per-line (no GROUP BY) via `bq query` heredoc, aggregate in Python; classify each SKU as floor vs variable; the floor-share ratio tells you whether the cost is structural (fleet size) or proportional (traffic).**\n","# GCP cost diagnosis from BQ billing export\n\nWhen the user asks \"why is GCP X expensive\" or \"drill into \u003Cservice> cost\" and the answer needs per-SKU granularity (not just the Cloud Console Subtotal), the BQ billing export is the source of truth. This skill encodes the **two non-obvious failure modes** that block the obvious approach and the working pattern that gets around them.\n\n## When this skill fires\n\n- \"Why is Cloud Run so expensive\" \u002F \"why did GCP cost spike\" \u002F \"drill into \u003Cservice> cost\"\n- The user shares a `[Daily GCP cost] YYYY-MM-DD — $X.XX` email (the worldarchitect daily cron template) and wants service breakdown\n- The user pastes a GCP cost table screenshot and asks about a specific service line\n- A spend-alert cron fires (see `spend-alert-bridge`) and the named mechanism points at a specific GCP service (Cloud Run, Cloud SQL, Cloud Storage, etc.)\n- Any question that requires per-SKU or per-service breakdown of GCP spend over a specific day or trend window\n\n**Anti-trigger:** if the user wants the **official billed figure** (for invoicing \u002F accounting), use the Cloud Console Subtotal directly. The BQ export runs **10-30% higher** than the Subtotal for ~2-3 weeks after a day closes because GCP keeps re-loading late-arriving usage events. State this caveat once when reporting BQ numbers.\n\n## The two failure modes\n\n### Failure mode 1 - `bq query` returns misleading \"Aggregations of aggregations are not allowed\"\n\nThe billing export schema has nested STRUCT fields (`service.description`, `sku.description`, `project.name`, `usage.amount`, `usage.unit`). When you write what looks like a normal GROUP BY query against this table:\n\n```sql\nSELECT sku.description AS sku, SUM(cost) AS cost\nFROM `project.billing_export.gcp_billing_export_v1_*`\nWHERE service.description = 'Cloud Run'\nGROUP BY sku\n```\n\n`bq query` returns:\n\n```\nError in query string: Error processing job ...: Aggregations of aggregations are not allowed at [9:12]\n```\n\nThe error message points at `SUM(cost)` but the query is syntactically valid (verified by running the SAME query via the python `google-cloud-bigquery` client - it works fine). The CLI parser hits some edge case with the heredoc \u002F STRUCT field combination.\n\n**The actual problem is not aggregations.** The query is fine. The CLI is misreporting.\n\n**The workaround that always works:**\n\n1. Drop the `GROUP BY` - select per-line items only.\n2. Pipe through `bq query --format=json`.\n3. Aggregate in Python (`collections.defaultdict` is fine - no `pandas` needed for thousands of rows).\n\n**The query that works (no GROUP BY):**\n\n```sql\nSELECT\n  sku.description AS sku,\n  usage.unit AS unit,\n  usage.pricing_unit AS pricing_unit,\n  cost,\n  usage.amount AS usage_qty\nFROM `PROJECT.billing_export.gcp_billing_export_v1_BILLINGACCT_HASH`\nWHERE usage_start_time >= TIMESTAMP('YYYY-MM-DD 00:00:00 UTC')\n  AND usage_start_time \u003C  TIMESTAMP('YYYY-MM-DD+1 00:00:00 UTC')\n  AND service.description = 'SERVICE_NAME'\nORDER BY cost DESC\nLIMIT 2000\n```\n\nThen aggregate in Python:\n\n```python\nimport subprocess, json\nfrom collections import defaultdict\n\nr = subprocess.run(\n    ['bq', 'query', '--use_legacy_sql=false', '--format=json', '--max_rows=2000'],\n    input=open('\u002Ftmp\u002Fq.sql').read(), capture_output=True, text=True, timeout=60,\n)\ndata = json.loads(r.stdout)\n\nagg = defaultdict(lambda: {'cost': 0.0, 'usage': 0.0, 'unit': '', 'lines': 0})\nfor row in data:\n    agg[row['sku']]['cost'] += float(row['cost'])\n    agg[row['sku']]['usage'] += float(row['usage_qty'])\n    agg[row['sku']]['unit'] = row.get('unit') or row.get('pricing_unit') or ''\n    agg[row['sku']]['lines'] += 1\n\ntotal = sum(v['cost'] for v in agg.values())\nfor sku, v in sorted(agg.items(), key=lambda x: -x[1]['cost']):\n    print(f\"{sku[:69]:\u003C70} {str(v['unit'])[:14]:\u003C15} ${v['cost']:>8.2f}\")\nprint(f\"TOTAL ${total:.2f}\")\n```\n\n### Failure mode 2 - `bq` subprocess can't find its `utils` module when launched from a sandbox\n\nIf you call `subprocess.run(['bq', 'query', ...])` from inside an `execute_code` sandbox without sourcing `~\u002F.bashrc`, the gcloud SDK's `utils` import fails with:\n\n```\nImportError: cannot import name 'bq_error' from 'utils'\n```\n\n**Fix:** source `~\u002F.bashrc` first (which sets up the gcloud SDK env), or run `bq query` from a `terminal()` tool call (which uses the user's normal shell).\n\n```bash\nbash -c 'source ~\u002F.bashrc 2>\u002Fdev\u002Fnull; bq query --use_legacy_sql=false --format=json \u003C \u002Ftmp\u002Fq.sql > \u002Ftmp\u002Fq.json 2>\u002Ftmp\u002Fq.err; echo exit=$?'\n```\n\nThe `terminal()` tool with bash wrapping is the reliable path.\n\n## The discovery pattern - find the table + project + billing account\n\nBefore any cost query, you need three identifiers. Get them in this order:\n\n```bash\n# 1. Project (default if not multi-project)\ngcloud config get-value project\n\n# 2. Billing dataset name (NOT always called \"billing\")\nbq ls -d --project_id=PROJECT\n# Look for: billing_export \u002F billing \u002F cloud_billing_export\n\n# 3. Full billing export table name (note the long hash suffix)\nbq ls DATASET\n# Pick the table starting with `gcp_billing_export_v1_` (not `_resource_v1_`)\n\n# 4. Confirm schema (column names are STRUCT - see Failure mode 1)\nbq show DATASET.gcp_billing_export_v1_HASH | head -50\n```\n\n**Critical detail:** The billing export table name has a long hex suffix (`gcp_billing_export_v1_011269_D08BDB_79D8F2`). It's tied to the **billing account**, not the project. If you support multiple billing accounts (e.g. dev vs prod), each one has its own table. Don't try to wildcard with `gcp_billing_export_v1_*` - the wildcard works in some clients and fails in `bq query` heredocs.\n\n## The diagnosis framework - cost-floor vs cost-variable\n\nOnce you have the per-SKU breakdown for a single day, classify each SKU into one of two buckets. This tells you whether the cost is structural (idle fleet) or proportional (traffic):\n\n| Bucket | Indicator SKUs | Diagnosis direction |\n|---|---|---|\n| **Floor** (idle baseline) | \"Min Instance Memory\", \"Min Instance CPU\", \"Idle VM time\", \"Always Free tier overage\" | These scale with **fleet size x uptime**, not traffic. If floor > 70% of service spend, the fix is **right-size \u002F scale-to-zero \u002F stop running excess services** |\n| **Variable** (per-request) | \"CPU (Request-based billing)\", \"Memory (Request-based billing)\", \"Requests\", \"Network egress\" | These scale with traffic. Diagnose with traffic dashboards \u002F recent deploy spike |\n\n**Rule of thumb:** if floor SKUs dominate, the cost is **structural** and the fix is at the service-config layer (Cloud Run min-instance, VM rightsizing, etc.). If variable SKUs dominate, the cost is **proportional** and the fix is at the application \u002F traffic layer.\n\n**Verified example (Cloud Run, project `worldarchitecture-ai`, 2026-07-16):**\n\n| SKU | Cost | Bucket |\n|---|---:|---|\n| Services Min Instance Memory | $8.93 | **floor** |\n| Services Min Instance CPU | $3.20 | **floor** |\n| Services CPU (Request-based) | $1.57 | variable |\n| Services Memory (Request-based) | $0.52 | variable |\n| Network egress + Jobs + Requests | $0.06 | variable |\n| **Total** | **$14.28** | **floor = $12.13 (85%)** |\n\nFloor dominance means the user should look at fleet right-sizing, not traffic. For this fleet, `mvp-site-app-s1` through `s10` (10 staging slots) plus `dev`\u002F`preview`\u002F`stable`\u002F`staging` (4 slots) at 4 vCPU \u002F 16 GiB each = ~14 large slots held warm. Downsizing dev\u002Fpreview\u002Fstaging would drop the floor by ~50%.\n\n## The trend query - confirm \"is this new or chronic?\"\n\nIf the user asks \"why did X spike today\" but actually the same floor has been there for weeks, you need a 7-day trend. Same query shape, but with `usage_start_time >= TIMESTAMP(YYYY-MM-DD-N 00:00:00 UTC)` for a window:\n\n```sql\nSELECT\n  DATE(usage_start_time) AS day,\n  ROUND(SUM(CASE WHEN sku.description LIKE 'Services Min Instance Memory%' THEN cost ELSE 0 END), 2) AS min_inst_mem,\n  ROUND(SUM(CASE WHEN sku.description LIKE 'Services Min Instance CPU%' THEN cost ELSE 0 END), 2) AS min_inst_cpu,\n  ROUND(SUM(CASE WHEN sku.description LIKE 'Services CPU (Request%' THEN cost ELSE 0 END), 2) AS req_cpu,\n  ROUND(SUM(CASE WHEN sku.description LIKE 'Services Memory (Request%' THEN cost ELSE 0 END), 2) AS req_mem,\n  ROUND(SUM(cost), 2) AS total\nFROM `PROJECT.billing_export.gcp_billing_export_v1_HASH`\nWHERE service.description = 'SERVICE_NAME'\n  AND usage_start_time >= TIMESTAMP('YYYY-MM-DD-7 00:00:00 UTC')\n  AND usage_start_time \u003C  TIMESTAMP('YYYY-MM-DD+1 00:00:00 UTC')\nGROUP BY day\nORDER BY day\n```\n\nIf the trend shows the **floor** is the same every day and **variable** is what spiked, the user has a traffic anomaly to investigate. If the floor itself is rising day-over-day, they have a fleet-size drift (new services coming online, autoscaler floor creeping up, etc.).\n\n## Per-service attribution - the labels gap\n\n**Heads up:** Cloud Run billing export rows do NOT carry a per-service label by default. The export has `project.labels`, `labels`, and `system_labels` arrays, but they're typically empty unless someone manually tagged each Cloud Run service with `gcloud run services update SERVICE --update-labels=service_name=SERVICE`.\n\nIf labels are present, you can attribute cost per service via:\n\n```sql\nSELECT\n  -- labels is REPEATED RECORD with key\u002Fvalue - extract via UNNEST\n  l.value AS service_label,\n  SUM(cost) AS cost\nFROM `PROJECT.billing_export.gcp_billing_export_v1_HASH`,\n     UNNEST(labels) AS l\nWHERE l.key = 'service_name'\n  AND service.description = 'Cloud Run'\n  AND usage_start_time >= ...\nGROUP BY service_label\nORDER BY cost DESC\n```\n\nIf labels are NOT present (the common case), the workaround is to **cross-reference with `gcloud run services list`** and reason about which services are running with what min-instance config, then map back to the SKU breakdown by memory\u002FCPU footprint. This is what we did in the verified example - found 14 services at 4 vCPU\u002F16 GiB by running `gcloud run services list --format=table(...)` then explained the $8.93 min-instance-memory line as that fleet's idle footprint.\n\n**Recipe for the gcloud inventory:**\n\n```bash\ngcloud run services list --project=PROJECT --region=REGION \\\n  --format=\"table(metadata.name,metadata.annotations['autoscaling.knative.dev\u002FminScale'],spec.template.spec.containers[0].resources.limits)\"\n```\n\nAdd a `MIN_SCALE` annotation column if you want to see which services have min-instance set (vs default 0 = scale-to-zero).\n\n## Pitfalls\n\n### Pitfall 1: Trusting `bq query` error messages at face value\n\nThe \"Aggregations of aggregations are not allowed\" error is a red herring. The query is fine. The CLI parser is misreporting. Always cross-check by running the same query through the Python client or by stripping the GROUP BY. If the no-GROUP-BY version returns rows and the GROUP BY version errors with that exact message, you're hitting the CLI bug - not a real syntax error.\n\n### Pitfall 2: Forgetting the late-arrival caveat\n\nThe BQ number runs **10-30% higher** than the Cloud Console Subtotal for 2-3 weeks after a day closes. When reporting the BQ number, state this caveat once. If the user is comparing against a Cloud Console screenshot, explain that:\n- BQ = forward-looking, accumulates late events\n- Cloud Console Subtotal = point-in-time, more authoritative for \"what got billed today\"\n- Both will converge once the invoice closes\n\n### Pitfall 3: Aggregating on `usage.unit` directly\n\n`usage.unit` (e.g. \"byte-seconds\") is the raw measurement unit. `usage.pricing_unit` (e.g. \"gibibyte\") is the billing unit. When computing per-unit cost, use `usage.amount_in_pricing_units` (the value scaled to pricing_unit), NOT `usage.amount` (raw). Otherwise you get nonsense rates.\n\n### Pitfall 4: Treating all Cloud Run spend as \"traffic-driven\"\n\nCloud Run's two SKU families look similar in name (\"Services Memory\" vs \"Services Min Instance Memory\") but bill completely differently. If you only see \"Cloud Run is $14\u002Fday\" without breaking down by SKU, you'll misdiagnose every cost-spike report. Always pull the per-SKU table before forming a hypothesis.\n\n### Pitfall 5: Running `bq query` from a subprocess without sourcing bashrc\n\nThe gcloud SDK's `bq` command depends on a `utils` import that breaks when the env doesn't have the SDK path. Symptom: `ImportError: cannot import name 'bq_error' from 'utils'`. Always wrap in `bash -c 'source ~\u002F.bashrc 2>\u002Fdev\u002Fnull; bq query ...'`.\n\n### Pitfall 6: Missing the cost-floor vs variable distinction when reporting\n\nIf you only report the per-SKU table without naming which SKUs are floor vs variable, the user has to do the analysis themselves. Always group SKUs into the two buckets and state the floor-share ratio. \"85% floor \u002F 15% variable\" is the single most useful sentence in any Cloud Run cost-diagnosis reply.\n\n## Reference and cross-links\n\n- `references\u002F2026-07-17-cloud-run-cost-breakdown.md` - session-specific evidence: full per-SKU table for `worldarchitecture-ai` Cloud Run on 2026-07-16, gcloud service inventory (14 services at 4 vCPU\u002F16 GiB), 7-day trend showing floor dominance, and the resulting recommended fix (downsize dev\u002Fpreview\u002Fs1-s10\u002Fstaging).\n- `~\u002F.hermes\u002Fskills\u002Fgh-actions-slow-runs\u002FSKILL.md` - sibling skill for GH Actions spend spikes (different export, different scope).\n- `~\u002F.hermes\u002Fskills\u002Fspend-alert-bridge\u002FSKILL.md` - verify-alert discipline; use this skill AFTER spend-alert-bridge confirms the alert is real.\n- `~\u002F.hermes\u002Fskills\u002Fworldarchitect\u002Fwa-cloud-run-deploy-failure-debug\u002FSKILL.md` - Cloud Run *deploy* failure diagnosis (NOT cost). Different concern, same project.\n- `~\u002F.hermes\u002Fskills\u002Fmac-di[REDACTED_OPENAI_KEY]\u002FSKILL.md` - the di[REDACTED_OPENAI_KEY] sibling skill (different resource, same \"first-use three concurrent lanes\" pattern).\n\n## One-line summary\n\n**Query per-line (no GROUP BY) via `bq query` heredoc, aggregate in Python; classify each SKU as floor vs variable; the floor-share ratio tells you whether the cost is structural (fleet size) or proportional (traffic).**\n",{},"main","42ccd02cbf5e8e47839dc3eab6fc754dc6b339cf","hermes\u002Fskills\u002Fdevops\u002Fgcp-cost-diagnosis-bq-billing-export",0,"https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills","https:\u002F\u002Fgithub.com\u002Fjleechanorg","2026-11-11 22:19:36",{"source":7,"skillName":5,"commands":37,"prompt":47,"zipAvailable":48},[38,41,44],{"tool":39,"command":40},"npx","npx skills add jleechanorg\u002Fjleechan-skills --skill gcp-cost-diagnosis-bq-billing-export",{"tool":42,"command":43},"bunx","bunx skills add jleechanorg\u002Fjleechan-skills --skill gcp-cost-diagnosis-bq-billing-export",{"tool":45,"command":46},"pnpm","pnpm dlx skills add jleechanorg\u002Fjleechan-skills --skill gcp-cost-diagnosis-bq-billing-export","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002Fhermes\u002Fskills\u002Fdevops\u002Fgcp-cost-diagnosis-bq-billing-export\u002FSKILL.md",true,[50,56],{"path":51,"resourceType":52,"resourceTypeDesc":53,"sizeBytes":54,"sourceHtmlUrl":55},"references\u002F2026-07-17-cloud-run-cost-breakdown.md","REFERENCE","参考文档",7607,"https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002Fhermes\u002Fskills\u002Fdevops\u002Fgcp-cost-diagnosis-bq-billing-export\u002Freferences\u002F2026-07-17-cloud-run-cost-breakdown.md",{"path":57,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":60,"sourceHtmlUrl":61},"scripts\u002Fdiagnose_gcp_cost.py","SCRIPT","脚本",4163,"https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002Fhermes\u002Fskills\u002Fdevops\u002Fgcp-cost-diagnosis-bq-billing-export\u002Fscripts\u002Fdiagnose_gcp_cost.py",[63,81,103,117,131,147],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":6,"repositoryNo":11,"starsCount":12,"licenseSpdxId":13,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":24},"01M716Y5MTJ6GYJZ2ATXCJK8G4","instinct","Use when the user invokes \u002Finstinct or \u002Fin, or asks to \"message Instinct\", \"ask Instinct\", \"tell Instinct\", \"check Instinct\", or read replies from the configured peer assistant. iMessage first, WhatsApp Web as fallback.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Finstinct\u002FSKILL.md",[],[65,70,71,72,73,74,75,76,77,78,79,80],"user","when","replies","check","use","identity","whatsapp","gmail","read","message","ask",{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":6,"repositoryNo":11,"starsCount":12,"licenseSpdxId":13,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":92,"publishedAt":24},"01M716Y4HF7FVVMJSA2CCVN5VF","hr","Compatibility alias for history-resume; produce the same fresh-state, provenance-aware resume briefing.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fhr\u002FSKILL.md",[87],{"categoryNo":88,"categoryName":89,"taxonomyType":90,"taxonomyTypeDesc":91},"01JBSEEDCAT000000000000038","人力资源","PROFESSION","适用职业",[83,93,94,95,96,97,98,99,100,101,102],"resume","compatibility","provenance","for","history","produce","alias","briefing","state","fresh",{"skillNo":104,"name":105,"nameZh":6,"description":106,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":6,"repositoryNo":11,"starsCount":12,"licenseSpdxId":13,"licenseFileUrl":6,"sourceHtmlUrl":107,"categories":108,"keywords":110,"publishedAt":24},"01M716Y1FFNFKFJYYKF8HQA6T8","history-resume","Resume briefing for the current repository branch or worktree using fresh state and admitted history sources.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fhistory-resume\u002FSKILL.md",[109],{"categoryNo":88,"categoryName":89,"taxonomyType":90,"taxonomyTypeDesc":91},[97,93,111,112,70,113,100,114,96,115,116,101,102],"history resume","branch","analysis","repository","worktree","sources",{"skillNo":118,"name":119,"nameZh":6,"description":120,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":6,"repositoryNo":11,"starsCount":12,"licenseSpdxId":13,"licenseFileUrl":6,"sourceHtmlUrl":121,"categories":122,"keywords":123,"publishedAt":24},"01M716XWXJ8EPHK4RH9T14TR85","grok","Alias for the canonical grokbot browser conversation workflow; \u002Fgrok talks to the Grok bot the same way \u002Fdot talks to Dot.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fgrok\u002FSKILL.md",[],[119,124,125,126,96,127,99,128,129,130],"dot","bot","workflow","canonical","conversation","browser","grokbot",{"skillNo":132,"name":133,"nameZh":6,"description":134,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":6,"repositoryNo":11,"starsCount":12,"licenseSpdxId":13,"licenseFileUrl":6,"sourceHtmlUrl":135,"categories":136,"keywords":137,"publishedAt":24},"01M70N9Q07DTE0EW5KFXMMMSJ0","codexa-bench","Compare `codexa` and normal `codex` on a fair split of comparable work by default; use `--strict` for duplicate runs of the exact same task.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fcodexa-bench\u002FSKILL.md",[],[138,139,140,74,141,96,142,143,144,145,146],"codexa bench","work","split","default","duplicate","task","compare","normal","codex",{"skillNo":148,"name":149,"nameZh":6,"description":150,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":6,"repositoryNo":11,"starsCount":12,"licenseSpdxId":151,"licenseFileUrl":152,"sourceHtmlUrl":153,"categories":154,"keywords":166,"publishedAt":24},"01M6C24S2X60SP8BBFGAEKPEGW","frontend-design","Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, artifacts, posters, or applications (examples include websites, landing pages, dashboards, React components, HTML\u002FCSS layouts, or when styling\u002Fbeautifying any web UI). Generates creative, polished code and UI design that avoids generic AI aesthetics.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ffrontend-design\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ffrontend-design\u002FSKILL.md",[155,160,163],{"categoryNo":156,"categoryName":157,"taxonomyType":158,"taxonomyTypeDesc":159},"01JBSEEDCAT000000000000009","前端开发","SCENARIO","使用场景",{"categoryNo":161,"categoryName":162,"taxonomyType":158,"taxonomyTypeDesc":159},"01JBSEEDCAT000000000000015","视觉设计",{"categoryNo":164,"categoryName":165,"taxonomyType":90,"taxonomyTypeDesc":91},"01JBSEEDCAT000000000000032","设计师",[167,168,169,74,170,171,172,70,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,71,191],"frontend","design","frontend design","generic","ui design","react","html","components","code","artifacts","ui","aesthetics","Production","web components","landing","applications","quality","web","css","create","examples","with","react components","web ui","pages"]