data-platform-observability
Build platform monitoring and SLO instrumentation for data pipelines: Databricks system tables (system.lakeflow jobs, system.billing.usage, system.access.audit, system.query.history), Databricks SQL alerts via the REST API, Microsoft Fabric Monitoring Hub and Capacity Metrics app, freshness/volume/quality SLO signals, and alert design that pages on symptoms rather than causes. Use when setting up data platform monitoring, job failure or cost alerting, Databricks or Fabric observability, pipeline SLOs, freshness checks, or on-call alert design.
- License 许可
- Apache-2.0
- 最近更新
- 2026-10-27
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- data-platform-observability
- description
- Build platform monitoring and SLO instrumentation for data pipelines: Databricks system tables (system.lakeflow jobs, system.billing.usage, system.access.audit, system.query.history), Databricks SQL alerts via the REST API, Microsoft Fabric Monitoring Hub and Capacity Metrics app, freshness/volume/quality SLO signals, and alert design that pages on symptoms rather than causes. Use when setting up data platform monitoring, job failure or cost alerting, Databricks or Fabric observability, pipeline SLOs, freshness checks, or on-call alert design.
03 SKILL.md 正文
Data Platform Observability
Outcome
Pipelines and platforms emit the signals needed to detect consumer-impacting failures, and every page names impact, owner, and first action — with concrete wiring for Databricks and Microsoft Fabric.
Setup
- Databricks: configure the CLI (
databricks configure --profile <name>); tokens come from theDATABRICKS_TOKENenv var or a secret store, never from files or prompts the agent writes. Verify withdatabricks --version(use GA 1.0.0+) anddatabricks current-user me. System tables require account-admin enablement per metastore and region; confirm access withSELECT * FROM system.information_schema.tables LIMIT 5on a SQL warehouse. - Fabric: Monitoring Hub is available to any user with item permissions (Monitor in the nav pane). The Capacity Metrics app must be installed by a capacity admin; its data lags 10-15 minutes. Fabric REST API calls need an Entra ID token (e.g.
az account get-access-token --resource https://api.fabric.microsoft.com). - Never log secrets, connection strings, or query result contents into artifacts; record metric values and IDs only.
Method
-
Define consumer-visible SLIs first: freshness (time since last successful load), volume (rows or bytes vs expected range), and quality-rule pass rate. Attach each to an SLO with a window, an owner, and an error budget before wiring any tooling.
-
Instrument the pipeline to emit results into a monitoring table: run status, duration, rows processed, watermark, and per-rule quality outcomes. Do not infer health from "the job succeeded."
-
Wire platform telemetry:
- Databricks jobs:
system.lakeflow.job_run_timelineandsystem.lakeflow.job_task_run_timelinefor run status, duration, and failure reasons across workspaces. - Databricks cost:
system.billing.usagegrouped by sku_name, usage_metadata.job_id, or warehouse_id; expect up to 12 hours latency. - Databricks security:
system.access.auditfor who changed jobs, tables, or permissions. - Fabric runs: Monitoring Hub (Activities page, filter Status = Failed; Historical runs for 30-day history per item).
- Fabric capacity: Capacity Metrics app — Health, Compute, and Timepoint pages for CU consumption, throttling, and the operations behind an overload.
- Databricks jobs:
-
Create alerts on the SLIs, not on infrastructure trivia. On Databricks, back an alert with a monitoring query on a SQL warehouse and create it through the Alerts REST API, e.g.:
curl -X POST "$DATABRICKS_HOST/api/2.0/alerts" \ -H "Authorization: Bearer $DATABRICKS_TOKEN" \ -d '{"name":"orders freshness breach","query_id":"<query-id>", "options":{"column":"minutes_since_load","op":">","value":"60"}, "rearm": 3600}'Confirm the current request shape against the Alerts API reference before sending; verify the created alert with the matching GET. Creating, updating, or deleting alerts is mutating — get user approval first. On Fabric, use scheduled-item failure notifications (Monitoring Hub → Schedule failures) and Fabric capacity overview events; the Capacity Metrics app itself does not support alerts.
-
Page on symptoms, not causes. A page means a consumer-facing SLO is burning: data is stale, volume is off, a quality rule failed. CPU spikes, queue depth, or a single retried task go to dashboards unless they already threaten the SLO. Every alert names the affected dataset and consumers (use lineage), the owner, the first check, and the runbook link; an alert without an operator action becomes a dashboard tile.
-
Verify end to end: force one stale-data, one volume-drop, and one failed-run scenario in a test workspace and confirm detection, routing, and deduplication. State the rollback: disable the alert (or delete via the API) and restore prior thresholds.
Permission boundaries
- Databricks system tables: account admin by default; grant least-privilege read via dynamic views per team. Alerts: CAN MANAGE on the alert to change it; the alert query runs on its warehouse's identity. Billing and audit data are account-wide — restrict to platform teams.
- Fabric: Monitoring Hub shows only items the caller can view; Capacity Metrics requires capacity admin to install and share; Fabric REST API calls run as the token's identity — use a service principal with read-only workspace roles for automation.
Documentation
- System tables reference: https://docs.databricks.com/aws/en/admin/system-tables/
- Jobs system tables (system.lakeflow): https://docs.databricks.com/aws/en/admin/system-tables/jobs
- Billing usage table: https://docs.databricks.com/aws/en/admin/system-tables/billing
- Audit log table: https://docs.databricks.com/aws/en/admin/system-tables/audit-logs
- Databricks Alerts REST API: https://docs.databricks.com/api/workspace/alerts/create
- Fabric Monitoring Hub: https://learn.microsoft.com/en-us/fabric/admin/monitoring-hub
- Fabric Capacity Metrics app: https://learn.microsoft.com/en-us/fabric/enterprise/metrics-app
- Databricks CLI: https://docs.databricks.com/aws/en/dev-tools/cli
Shared resources
Load only the resources needed for the current task. The links resolve inside the current agent package.
../../references/reliability-and-slo-guide.md../../references/data-quality-dimensions.md../../templates/data-product-slo.yaml
Troubleshooting
- System tables return nothing: they must be enabled per metastore by an account admin, are regional, and have latency (about 1 hour for query history, 12 hours for billing); verify the metastore and window before assuming no data.
- Alert fires but query history shows success: the alert ran on stale data or the wrong warehouse; check the alert query's own record in
system.query.historyand its warehouse. - Billing totals do not match invoice: aggregate
usage_quantityincluding RETRACTION and RESTATEMENT record types, and allow 12+ hours of lag. - Monitoring Hub missing a run: it shows up to 100 recent activities over 30 days; use Historical runs for the item or the workspace's item job API.
- Capacity Metrics app shows old data: usage lags 10-15 minutes and new items appear only after the next scheduled semantic model refresh.
- Fabric capacity throttling with no clear culprit: use the Timepoint page to break down the 30-second window by operation, then correlate with Monitoring Hub activities at that time.
Completion check
The work is done when each critical pipeline has named SLIs and SLO targets, platform telemetry is queryable (system tables or Fabric hub), at least one symptom-based alert was proven to fire and route in a forced-failure test, and every page identifies impact, owner, and first action.
04 同仓库更多 Skills
twenty-year-land-plan
Apply the Bren discipline of evaluating every land and development decision on a twenty-year horizon, with a comprehensive plan for how the land, the community, and the surrounding public infrastructure evolve together. Deals optimized for a five-year exit are structurally different from deals that compound across generations; insist on knowing which kind you are making. Use when evaluating land, entitlements, master-planned development, or phasing decisions, and when separating five-year exit deals from generational holds.
thematic-gut-check
Apply the Gray discipline of identifying the structural theme behind every real estate deal, testing where that theme sits in its capital-flow lifecycle, and insisting that deals without a named theme — or deals late in a crowded theme — clear a higher bar. A mediocre deal in a strong theme usually beats a clever deal in a weak one. Use when naming the structural thesis behind a deal, judging whether that theme is early or crowded, and setting the bar the deal must clear.
supply-pipeline-gut-check
Apply the Zell discipline of grounding every real estate thesis in the local supply pipeline. Demand counts over adjectives — units under construction, units permitted, units proposed — and convert them into a forward absorption schedule. A deal that ignores the pipeline is a bet on a market you have not actually studied. Use when validating a market thesis, forecasting absorption and rent growth, or counting units under construction, permitted, and proposed.
ruin-scenario-mapping
Apply the Taleb discipline of identifying, for every real estate deal, the specific chain of events that produces ruin — permanent, unrecoverable capital loss — and verifying that no single such chain has non-trivial probability. Deals with any plausible ruin path must be restructured or abandoned. Use when sizing leverage, testing refinancing, covenant, and liquidity risk, or checking whether any single failure chain can permanently wipe out the position.
revenue-cost-architecture
Design a venture's revenue streams, pricing logic, cost structure, and unit economics so the business model has a coherent theory of value capture, not just value creation. Use after the value proposition and customer segment are set and before financial projections, when choosing a pricing model, calculating contribution margin or CAC payback, or checking whether the unit economics hold at scale.
replacement-cost-stress-test
Apply the Buffett discipline of pricing every real estate asset against the cost to rebuild it from scratch. The ratio of purchase price to replacement cost sets a ceiling on long-run downside: below replacement cost, new supply cannot profitably undercut you; at or above it, you are buying a liability disguised as an asset. Use when setting a purchase price, judging downside protection, or testing how exposed an asset is to newly built competing supply.