AGENT SKILL

app-observability

Get RED metrics + service maps + frontend RUM + AI/LLM monitoring out of Grafana Cloud — Application Observability (`traces_spanmetrics_*` from OTel traces, p50/p95/p99 latency, exemplar-to-trace, traces-to-logs / profiles), Frontend Observability with the Faro Web SDK (Core Web Vitals, session replay, `pushError`, React + router integration, `TracingInstrumentation` for browser → backend trace correlation), and AI Observability via OpenLIT (token / cost / latency, GPU, hallucination + toxicity evals). Use when standing up APM for a service, wiring an Alloy OTLP receiver + forwarding to Cloud, instrumenting a React frontend for RUM, debugging why service-map edges are missing, monitoring LLM cost drift, or correlating a frontend error to its backend trace — even when the user says "set up APM", "show service map", "monitor browser perf", "session replay", "RUM SDK", or "watch our OpenAI bill" without naming App / Frontend / AI Observability.

License 许可
Apache-2.0
最近更新
2026-11-14
Stars
★ 12
Forks
5

01 前言 Front Matter

name
app-observability
license
Apache-2.0
description
Get RED metrics + service maps + frontend RUM + AI/LLM monitoring out of Grafana Cloud — Application Observability (`traces_spanmetrics_*` from OTel traces, p50/p95/p99 latency, exemplar-to-trace, traces-to-logs / profiles), Frontend Observability with the Faro Web SDK (Core Web Vitals, session replay, `pushError`, React + router integration, `TracingInstrumentation` for browser → backend trace correlation), and AI Observability via OpenLIT (token / cost / latency, GPU, hallucination + toxicity evals). Use when standing up APM for a service, wiring an Alloy OTLP receiver + forwarding to Cloud, instrumenting a React frontend for RUM, debugging why service-map edges are missing, monitoring LLM cost drift, or correlating a frontend error to its backend trace — even when the user says "set up APM", "show service map", "monitor browser perf", "session replay", "RUM SDK", or "watch our OpenAI bill" without naming App / Frontend / AI Observability.

02 资源文件 6

03 SKILL.md 正文

Grafana Cloud Application Observability

Docs: https://grafana.com/docs/grafana-cloud/monitor-applications/

Three products that share the same OTLP + Mimir / Loki / Tempo / Pyroscope plumbing:

  1. Application Observability — APM from OTel spanmetrics
  2. Frontend Observability — Faro Web SDK, RUM + session replay
  3. AI Observability — LLM / vector-DB monitoring via OpenLIT

Prerequisites

  • Grafana Cloud stack + OTLP endpoint + numeric instance ID + API key with MetricsPublisher + LogsPublisher + TracesPublisher
  • For APM: app instrumented with OTel SDK; for Frontend: a web app + Faro app key; for AI: Python ≥ 3.10
  • Grafana Alloy as the local OTLP receiver (recommended)

Common Workflows

1. Stand up APM — Alloy receiver → Grafana Cloud + verify

# 1. Set Cloud creds + start Alloy with config from references/apm.md
export GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
export GRAFANA_CLOUD_INSTANCE_ID=123456
export GRAFANA_CLOUD_API_KEY=glc_eyJ...
alloy fmt /etc/alloy/config.alloy   # syntax check
alloy run /etc/alloy/config.alloy

# 2. Verify Alloy is receiving + forwarding
curl -s http://localhost:12345/api/v0/web/components \
  | jq '.[] | select(.id|test("otelcol\\.exporter\\.otlphttp"))
        | {id, health:.health.state}'
# Expect health.state == "healthy"
curl -s http://localhost:12345/metrics \
  | grep -E 'otelcol_(receiver_accepted_spans|exporter_sent_spans)'

# 3. Point your app at Alloy (with required attributes!)
export OTEL_SERVICE_NAME="my-api"
export OTEL_RESOURCE_ATTRIBUTES="service.namespace=myteam,deployment.environment=production"
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

# 4. Verify spans landed in Tempo + spanmetrics generated
#    Tempo (TraceQL):  { resource.service.name = "my-api" }
#    Mimir (PromQL):   sum by (job) (rate(traces_spanmetrics_calls_total{service_name="my-api"}[5m]))
#    Expect > 0 within ~1 minute.

# 5. Verify it's wired to App Observability
#    Grafana → Application → Service Inventory: "my-api" should appear with RED metrics
#    Click into it → Service Map edges visible (requires span.kind on outbound calls)

Full Alloy block + required resource attributes + spanmetric names + correlation links: references/apm.md.

2. Instrument a React frontend with Faro

# 1. Install
npm install @grafana/faro-react @grafana/faro-web-tracing
// 2. initializeFaro with TracingInstrumentation + ReactIntegration (see references/faro.md)
//    Push a smoketest event so we have a known signal:
faro.api.pushEvent('faro_smoketest', { ts: Date.now().toString() });
# 3. Verify in DevTools Network — POST to /collect returns 202
#    (401 → wrong app key; 404 → wrong url region)

# 4. Verify in Grafana Cloud
#    - Frontend Observability → your app → Sessions: your session appears
#    - LogQL on Loki: {kind="event"} |= "faro_smoketest"
#    - With TracingInstrumentation: open the session → the trace ID links to Tempo

Full React example, CDN setup, session config: references/faro.md.

3. Add AI / LLM observability

pip install openlit==1.45.0
# At app startup
import openlit
openlit.init(application_name="my-ai-app", environment="production")
# Your existing OpenAI / Anthropic / Cohere calls now emit OTel spans + metrics.
# Env (same OTLP endpoint as APM)
export OTEL_SERVICE_NAME="my-ai-app"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-gateway-<region>.grafana.net/otlp"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(echo -n $ID:$KEY | base64)"

# Verify after a few LLM calls:
#   PromQL: sum by (gen_ai_request_model) (rate(gen_ai_usage_input_tokens_total[5m]))
#   Dashboard: Grafana → AI Observability → "GenAI Observability" auto-populates

Full OpenLIT install, evals/guards, GenAI metric list, dashboard names: references/ai-observability.md.

Full-stack correlation cheat sheet

Signal Product Storage Query
RED metrics App Observability Mimir PromQL
Traces Tempo Tempo TraceQL
Logs Loki Loki LogQL
Profiles Pyroscope Pyroscope ProfileQL
Browser RUM Frontend Observability Loki + Tempo LogQL / TraceQL
LLM metrics AI Observability Mimir PromQL

Correlation keys: service.name joins all signals; trace exemplars embed trace IDs in metric points; traceID in logs and traceparent injected by Faro for FE → BE linking.

Troubleshooting

  • Service missing from Service Inventory → missing service.namespace (job label) or deployment.environment resource attribute
  • Service Map edges missing → span.kind not set on outbound calls (must be CLIENT) or inbound (SERVER)
  • Faro /collect returns 401 → wrong app key; 404 → region in URL doesn't match the Faro app
  • No GenAI metrics → confirm OpenLIT version matches OTel semantic-conv version expected by Cloud; verify auth with curl as in workflow #3

References

Resources

04 同仓库更多 Skills

stanfish06
stanfish06/skillquarium

zarr-python

Stores and queries chunked N-D scientific arrays with Zarr-Python 3, including codecs, sharding, S3/GCS storage, and NumPy/Dask/Xarray integration. Use for array layout, bounded I/O, format migration, or scientific metadata preservation.

★ 12
stanfish06
stanfish06/skillquarium

yeet

Publish local changes to GitHub by confirming scope, committing intentionally, pushing the branch, and opening a draft PR through the GitHub app from this plugin, with `gh` used only as a fallback where connector coverage is insufficient.

★ 12
stanfish06
stanfish06/skillquarium

xarray

N-dimensional labeled arrays and datasets in Python. Built on top of NumPy and Dask. It introduces labels in the form of dimensions, coordinates, and attributes on top of raw NumPy-like arrays, making data analysis in physical sciences more intuitive and less error-prone. Use for working with multi-dimensional scientific data, NetCDF/GRIB/Zarr files, climate/weather/oceanographic datasets, remote sensing, geospatial imaging, large out-of-memory datasets with Dask, and labeled array operations.

★ 12
stanfish06
stanfish06/skillquarium

writing-mstest-tests

Use when asked to fix, rewrite, update, improve, modernize, show corrected code for, or explain existing MSTest tests or MSTest-specific configuration. Use for "review" when corrected code or edits are wanted, even for one pasted assertion or passing tests with bad failure output. Covers expected/actual labels; generic Boolean, collection, string, numeric, null, identity, exception, hard-cast, and object[] checks; TestContext/lifecycle; timeout/cancellation; OS/CI conditions, retry, cleanup, parallelization, MSTest.Sdk project setup, and MSTESTxxxx. Honor the installed MSTest version. DO NOT USE to design new test cases (code-testing), perform report-only audits, create project files rather than explain MSTest setup, run tests, migrate frameworks, or handle non-MSTest/non-.NET code.

★ 12
stanfish06
stanfish06/skillquarium

wgs-prs

End-to-end WGS to polygenic risk score pipeline. Takes paired-end FASTQ files (or a pre-existing VCF) through nf-core/sarek for variant calling, applies VCF QC (normalisation, hard filtering, Ti/Tv and Het/Hom checks), then computes polygenic risk scores via the PGS Catalog. Fills the FASTQ to VCF gap upstream of the gwas-prs skill.

★ 12
stanfish06
stanfish06/skillquarium

webapp-testing

Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.

★ 12