AGENT SKILL

openevidence-observability

Set up comprehensive observability for OpenEvidence integrations with metrics, traces, and alerts. Use when implementing monitoring for clinical AI operations, setting up dashboards, or configuring alerting for healthcare application health. Trigger with phrases like "openevidence monitoring", "openevidence metrics", "openevidence observability", "monitor openevidence", "openevidence alerts".

License 许可
MIT
最近更新
2026-05-26
Stars
★ 0
Forks
0

01 前言 Front Matter

name
openevidence-observability
author
Jeremy Longshore <jeremy@intentsolutions.io>
license
MIT
version
1.0.0
description
Set up comprehensive observability for OpenEvidence integrations with metrics, traces, and alerts. Use when implementing monitoring for clinical AI operations, setting up dashboards, or configuring alerting for healthcare application health. Trigger with phrases like "openevidence monitoring", "openevidence metrics", "openevidence observability", "monitor openevidence", "openevidence alerts".
allowed-tools
Read, Write, Edit
compatible-with
claude-code, codex, openclaw

02 资源文件 1

03 SKILL.md 正文

OpenEvidence Observability

Table of Contents

Overview

Set up comprehensive observability for OpenEvidence clinical AI integrations with Prometheus metrics, OpenTelemetry tracing, structured logging with PHI redaction, and Grafana dashboards.

Prerequisites

  • Prometheus or compatible metrics backend
  • OpenTelemetry SDK installed
  • Grafana or similar dashboarding tool
  • AlertManager or PagerDuty configured

Key Metrics

Metric Type Alert Threshold
openevidence_requests_total Counter N/A
openevidence_request_duration_seconds Histogram P95 > 15s
openevidence_errors_total Counter > 5% error rate
openevidence_cache_hits_total Counter < 50% hit rate
openevidence_rate_limit_remaining Gauge < 10% headroom
openevidence_deepconsult_active Gauge > 50 concurrent

Instructions

Step 1: Set Up Prometheus Metrics

Register counters (requests, errors, cache hits/misses), histograms (request duration, DeepConsult duration), gauges (rate limit remaining), and summaries (confidence scores) with appropriate labels.

Step 2: Instrument Client Wrapper

Wrap OpenEvidence client with metrics collection, cache hit/miss tracking, rate limit monitoring from response headers, and confidence score recording.

Step 3: Configure Distributed Tracing

Initialize OpenTelemetry with Google Cloud Trace exporter, HTTP and Express instrumentations, and service metadata.

Step 4: Set Up Structured Logging

Use pino with PHI redaction (patient.*, patientId, mrn, *.ssn) and OpenEvidence-specific child logger for clinical queries and DeepConsult events.

Step 5: Define Alert Rules

Create Prometheus alerts for: high error rate (>5% warning, >20% critical), high P95 latency (>15s), low cache hit rate (<50%), rate limit warning (<10 remaining), service down.

Step 6: Build Grafana Dashboard

Create panels for request rate, error rate gauge, latency heatmap, cache hit rate timeseries, rate limit gauge, and confidence score distribution.

Output

  • Prometheus metrics collection with /metrics endpoint
  • Instrumented client wrapper with tracing
  • Distributed tracing (OpenTelemetry + Cloud Trace)
  • Structured logging with PHI redaction
  • Alert rules for critical conditions
  • Grafana dashboard JSON

Error Handling

Issue Detection Resolution
Metrics endpoint down Prometheus scrape failure Check /metrics route registration
Missing traces No spans in Cloud Trace Verify OpenTelemetry SDK initialization
PHI in logs Audit review Add patterns to pino redact config
Alert fatigue Too many alerts Adjust thresholds, add for durations

Examples

Metrics Endpoint

router.get('/metrics', async (req, res) => {
  res.set('Content-Type', registry.contentType);
  res.send(await registry.metrics());
});

See detailed implementation for advanced patterns.

Resources

04 同仓库更多 Skills

SamuelChien
SamuelChien/mega-skills-collection

zustand-store-ts

Create Zustand stores with TypeScript, subscribeWithSelector middleware, and proper state/action separation. Use when building React state management, creating global stores, or implementing reactive state patterns with Zustand.

★ 0
SamuelChien
SamuelChien/mega-skills-collection

zustand-store-creator

Zustand Store Creator - Auto-activating skill for Frontend Development. Triggers on: zustand store creator, zustand store creator Part of the Frontend Development skill category.

★ 0
SamuelChien
SamuelChien/mega-skills-collection

zero-trust-config-helper

Zero Trust Config Helper - Auto-activating skill for Security Advanced. Triggers on: zero trust config helper, zero trust config helper Part of the Security Advanced skill category.

★ 0
SamuelChien
SamuelChien/mega-skills-collection

zapier-integration-helper

Zapier Integration Helper - Auto-activating skill for Business Automation. Triggers on: zapier integration helper, zapier integration helper Part of the Business Automation skill category.

★ 0
SamuelChien
SamuelChien/mega-skills-collection

zai-cli

Execute z.AI CLI providing vision, search, reader, and GitHub exploration via CLI and MCP. Use when user needs image/video analysis, OCR, UI-to-code conversion, error diagnosis, real-time web search, web page to markdown extraction, or GitHub code exploration. Trigger with phrases like "analyze this image", "search the web for", "read this page", "explore this repo", or "use zai". Requires Z_AI_API_KEY.

★ 0
SamuelChien
SamuelChien/mega-skills-collection

yt-research

Research competitor YouTube channels, niches, and trending topics for your content strategy. Use this skill whenever the user says "research channels", "analyze competitors", "find trending topics", "niche analysis", "competitive research", "what are other creators doing", "scrape YouTube channels", or wants to understand the competitive landscape for a specific tool or topic area. Use when working with yt research. Trigger with 'yt', 'research'.

★ 0