agento11y

OpenCode Prompt: Agent Observability Instrumentation

You are running in Opencode with repository files and shell access.

• Prefer direct file edits over speculative refactors.
• Before proposing broad changes, confirm impact scope with quick evidence.

Agent Observability Agent-First Instrumentation Brief

You are acting as a coding agent inside this repository. Your goal is to add or improve Grafana Agent Observability instrumentation with minimal, safe changes.

Mission

1. Find AI generation and tool/agent execution paths.
2. Add agento11y instrumentation using the local language SDK where possible.
3. Preserve behavior and keep diffs small.
4. Add or update tests for changed instrumentation behavior.
5. Explain what was instrumented and why.

For application instrumentation, use the Grafana Cloud API URL from Agent Observability configuration and Cloud basic auth. Do not invent Cloud or development endpoints. Coding-agent capture also supports the shared launcher's local mode, without Cloud credentials or Tempo.

Output contract (required)

Return:

• Top opportunities first (highest traffic / highest impact)
• For each opportunity:
  • exact file path(s)
  • why this location matters
  • concrete diff proposal
  • test plan
  • any risk or compatibility concern

Agent Observability architecture and ingest model (must follow)

• Agent Observability uses generation-first ingest:
  • gRPC: agento11y.v1.GenerationIngestService.ExportGenerations
  • HTTP parity: POST /api/v1/generations:export
• Traces/metrics go through OTEL collector/alloy, not through Agent Observability ingest.
• Required generation modes:
  • non-stream: SYNC
  • stream: STREAM
• Raw provider artifacts are default OFF and only enabled for explicit debug opt-in.

Authoritative references in the grafana/agento11y repo:

• docs/concepts/content-capture-modes.md
• docs/concepts/tags-and-metadata.md
• docs/concepts/tool-call-vs-tool-execution.md

OTel setup (required)

The agento11y SDK internally emits OTel spans and metrics (gen_ai.client.operation.duration, gen_ai.client.token.usage, gen_ai.client.time_to_first_token, gen_ai.client.tool_calls_per_operation). Without a configured TracerProvider and MeterProvider these go to the default no-op and are silently lost.

The SDK does NOT create OTel providers — that is the application's responsibility. Always ensure the app configures providers BEFORE creating the agento11y client, and shuts them down AFTER agento11y.shutdown().

Traces and metrics can be sent to Grafana Cloud in two ways. Always use env vars (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS) so the app doesn't hardcode assumptions.

Option A — Direct to Grafana Cloud (no collector needed)

Send OTLP straight to the Grafana Cloud OTLP gateway. The exact URL is stack-specific — get it from the Grafana Cloud portal → stack Details page (docs). Authentication uses Basic auth with instance ID and a Cloud API token.

Env vars:

OTEL_EXPORTER_OTLP_ENDPOINT=https://<your-otlp-gateway-url>   # from Cloud portal
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <base64(instance_id:cloud_api_token)>

The OTel SDK exporters read these env vars automatically — no extra code needed beyond the provider setup below.

Option B — Via Alloy / OTel Collector (optional)

Run Alloy or an OTel Collector in the user's deployment and forward telemetry to Grafana Cloud with credentials. Useful for centralized token management, retries, relabeling, and metadata enrichment.

Env vars:

OTEL_EXPORTER_OTLP_ENDPOINT=<your-collector-otlp-endpoint>

Provider setup (required for both options)

The snippets below configure TracerProvider and MeterProvider using OTLP/HTTP exporters that honour the env vars above.

Python

from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter

resource = Resource.create({"service.name": "my-app"})

tp = TracerProvider(resource=resource)
tp.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(tp)

mp = MeterProvider(resource=resource, metric_readers=[
    PeriodicExportingMetricReader(OTLPMetricExporter())
])
metrics.set_meter_provider(mp)
# Deps: opentelemetry-sdk, opentelemetry-exporter-otlp-proto-http

Go

traceExp, _ := otlptracehttp.New(ctx)
tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(traceExp), sdktrace.WithResource(res))
otel.SetTracerProvider(tp)
defer tp.Shutdown(ctx)

metricExp, _ := otlpmetrichttp.New(ctx)
mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricExp)), sdkmetric.WithResource(res))
otel.SetMeterProvider(mp)
defer mp.Shutdown(ctx)

JS/TS

import { metrics } from '@opentelemetry/api';
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';

const tp = new NodeTracerProvider({ resource });
tp.addSpanProcessor(new BatchSpanProcessor(new OTLPTraceExporter()));
tp.register();

const mp = new MeterProvider({
  resource,
  readers: [new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter() })],
});
metrics.setGlobalMeterProvider(mp);

Telemetry fields to prioritize

On generation and tool spans, capture or preserve these when available:

• identity and routing:
  • gen_ai.operation.name
  • agento11y.generation.id
  • gen_ai.conversation.id
  • gen_ai.agent.name
  • gen_ai.agent.version
  • agento11y.generation.parent_generation_ids
  • agento11y.sdk.name
• model:
  • gen_ai.provider.name
  • gen_ai.request.model
  • gen_ai.response.model
• request controls:
  • gen_ai.request.max_tokens
  • gen_ai.request.temperature
  • gen_ai.request.top_p
  • agento11y.gen_ai.request.tool_choice
  • agento11y.gen_ai.request.thinking.enabled
  • agento11y.gen_ai.request.thinking.budget_tokens
• usage and outcomes:
  • gen_ai.usage.input_tokens
  • gen_ai.usage.output_tokens
  • gen_ai.usage.cache_read_input_tokens
  • gen_ai.usage.cache_write_input_tokens
  • gen_ai.usage.reasoning_tokens
  • gen_ai.response.finish_reasons
  • error classification fields (error.type, error.category)

Tracing skill loads, resource reads, and commands

The SDKs and Go otelgenai support the OpenTelemetry conventions for loading skills, reading skill resources, and running commands. These conventions come from GenAI #498 and have Development status. The global schema declaration remains on the legacy schema while other signals still use it. Each execution creates one INTERNAL span with gen_ai.operation.name=execute_tool. The execution kind selects span naming and standard gen_ai.skill.* and process.* attributes. The SDK does not emit the execution kind as a span attribute. For the three recognized kinds, the five SDK clients add metric-only agento11y.tool.execution.kind to gen_ai.client.operation.duration. They also include the trimmed, nonblank gen_ai.skill.name supplied at start, including failed attempts and reduced capture modes. A recognized kind without a name retains only the kind; unset or unknown kinds add neither label. No other skill or process dimensions are added automatically. Standalone Go otelgenai automatic dimensions remain unchanged. Skill names remain under metric retention; use stable, nonsensitive names. See client metrics for query labels and cardinality.

Execution kind	Use	Suffix after execute_tool {tool}
load_skill	Load skill instructions.	Skill name when present.
read_skill_resource	Read a skill resource.	Skill plus resource; resource alone; otherwise none.
command	Execute commands, with or without a skill.	Skill plus resource; resource alone; executable name; otherwise none.

Resource reads and commands never append a skill name alone. Resource naming takes precedence over executable naming. Select Go ToolExecutionStart.ExecutionKind, Python execution_kind with ToolExecutionKind, or JS/TS executionKind with the string values above. Go otelgenai.Invocation uses ExecutionKind; Java uses setExecutionKind(...); .NET uses ExecutionKind. ToolType remains a category such as function or extension. Unset or unknown execution kinds remain generic and omit skill and process attributes, even when those fields are supplied. Do not infer execution kinds from tool names, arguments, paths, or results. SDK support does not add automatic plugin detection. Generic convenience loops stay generic and infer no exit codes.

Capture applicable fields only:

• All three: gen_ai.skill.name, gen_ai.skill.description, gen_ai.skill.source.uri.
• Resource reads and commands: also gen_ai.skill.resource.name.
• Commands only: also process.executable.name, process.executable.path, and completion-time process.exit.code.

Only populate process fields for an actual, directly executed single process, not a guessed executable or aggregated pipeline outcome. Report nullable integer exit codes at completion (Go *int64): zero is reported success; absent is unknown. Exit codes do not map to errors, span status, or error.type. Keep using the execution-error API. Applicable structural names reach samplers at creation; completion exit codes and typed private fields do not. Use the standard skill-name key, not agento11y.skill.name. A skill name alone does not identify a load. See the API field table for language spellings.

Typed skill and process fields follow the effective capture mode. metadata_only and full_with_metadata_spans exclude skill description, source URI, and executable path. Go/Python experimental OTel generation export resolves full_with_metadata_spans to full, so those clients can export private fields. Skill source URIs exported from typed fields lose all user information, query parameters, and fragments. Malformed values are omitted without filesystem/network lookups. In Go otelgenai, raw Invocation.Attributes and attributes returned by end hooks bypass capture filtering and URI scrubbing. They can override typed attributes. Callers must filter private attributes and sanitize source URIs themselves. Core recorders can retain private inputs. JS/Java completed debug history defaults to disabled (debugBufferSize: 0). To retain completed records in debug snapshots, use JS new Agento11yClient({ debugBufferSize: 64 }) or Java new Agento11yClientConfig().setDebugBufferSize(64). No environment variable sets debugBufferSize; verbose debug and AGENTO11Y_DEBUG do not control history. The client retains the newest debugBufferSize entries of each record type across all conversations. Snapshots return those entries oldest first by recording order, including after shutdown. Queue counters remain available with history disabled. Retention does not change exports, queue limits, spans, metrics, or capture policy. When enabled, tool snapshots retain raw arguments, results, and error text even in reduced modes. Arguments and results still reference the original payload objects. The capacity counts records, not bytes. Active recorders, pending exports, and caller-held snapshots can retain additional data. The no_tool_content arguments/results opt-in is unchanged and does not bypass reduced effective modes. See capture-mode privacy.

The shared local receiver stores optional skill/resource/executable names and int64 exit codes in tool-span sidecars, never the three private attributes. It prefers nonblank gen_ai.skill.name and accepts legacy agento11y.skill.name. Local Analytics > Skills counts retained marked calls, including loads, resource reads, commands, and failures. Tools and Skills overlap; never add their counts together. See Local skill analytics for exact identifiers, duration coverage, and session drilldowns. Older recordings without skill markers stay unmarked; history import does not reconstruct missing skill metadata. Local recording and analytics need no Tempo.

Live recognition supports Claude Code's native Skill, Pi's catalog-matched instruction reads, and OpenCode's standard skill tool, not arbitrary resource reads or commands. OpenCode retains the requested identifier, not a resolved directory basename, trimming only surrounding whitespace. A recognized attempt can fail and does not explain later activity. See OpenCode recognition limits. Cloud skill visibility requires retained tool and generation spans in Tempo plus deployed Sigil consumer support.

Multi-agent dependency tracking

When instrumenting multi-agent pipelines where one agent's output feeds into another:

• Set parent_generation_ids on the GenerationStart/seed with the generation ID(s) of the upstream agent(s) whose output this generation consumes.
• This is a list: a generation can depend on multiple parents (fan-in).
• Agent Observability uses these links to build a dependency DAG and propagate quality signals: if an upstream generation fails evaluation, all downstream dependents are flagged.

Example: an orchestrator spawns agents A, B, C where C depends on A and B:

• A: parent_generation_ids = [] (no parents)
• B: parent_generation_ids = [] (no parents)
• C: parent_generation_ids = [A.generation_id, B.generation_id]

SDK locations and how to instrument

Prefer these existing SDKs and wrappers before inventing custom plumbing:

• Go core SDK: go/ (see go/README.md)
  • StartGeneration, StartStreamingGeneration, StartToolExecution, StartEmbedding
• JS/TS SDK: js/ (see js/README.md)
  • startGeneration, startStreamingGeneration, startToolExecution, startEmbedding
• Python SDK: python/ (see python/README.md)
  • start_generation, start_streaming_generation, start_tool_execution, start_embedding
• Java SDK: java/ (see java/README.md)
  • startGeneration, startStreamingGeneration, withGeneration, withToolExecution
• .NET SDK: dotnet/ (see dotnet/README.md)
  • StartGeneration, StartStreamingGeneration, StartToolExecution, StartEmbedding

Provider wrappers and framework adapters already exist; reuse them where possible:

• Go providers: go-providers/openai, go-providers/anthropic, go-providers/gemini
• Python providers: python-providers/*
• Java providers: java/providers/*
• .NET providers: dotnet/src/Grafana.Agento11y.*
• Framework adapters:
  • Python: python-frameworks/*
  • Go Google ADK: go-frameworks/google-adk
  • Java Google ADK: java/frameworks/google-adk
  • JS subpath adapters documented in js/README.md

Useful repo examples to copy patterns from

• Go explicit generation flow:
  • go/agento11y/example_test.go
• Go provider wrapper examples:
  • go-providers/openai/sdk_example_test.go
  • go-providers/anthropic/sdk_example_test.go
  • go-providers/gemini/sdk_example_test.go
• .NET end-to-end emitter:
  • dotnet/examples/Grafana.Agento11y.DevExEmitter/Program.cs
• JS transport and framework behavior:
  • js/test/client.transport.test.mjs
  • js/test/frameworks.vercel-ai-sdk.test.mjs
• Python framework integration tests:
  • python-frameworks/*/tests/*.py

Implementation rules

• Keep behavior unchanged except instrumentation additions/fixes.
• Prefer small targeted patches over refactors.
• Use existing conventions in each language package.
• Keep raw artifacts disabled unless explicitly asked.
• Ensure non-stream wrappers set SYNC, stream wrappers set STREAM.
• Ensure lifecycle flush/shutdown semantics are preserved.
• When calling set_result / SetResult, always include all available fields:
  • response_id (provider correlation, maps to gen_ai.response.id)
  • response_model (actual model used)
  • stop_reason / finish_reason
  • Full token usage including cache_read_input_tokens, cache_write_input_tokens, and reasoning_tokens when the provider returns them
• Always check rec.err() / Err() after the generation recorder closes — SDK validation or enqueue errors are otherwise silent.
• Use tags on GenerationStart for filtering in the Agent Observability UI (e.g. pipeline name, layer, agent role).

Validation checklist

After proposing edits, include checks for:

• span attributes emitted as expected
• generation payload shape valid for ingest contract
• no regressions in existing tests
• language-specific tests or focused test additions for new instrumentation logic

Deliverable format (strict)

Provide:

1. Prioritized instrumentation opportunities
2. Proposed diffs per opportunity
3. Test updates per opportunity
4. Rollout/risk notes

If no safe opportunities are found, explain exactly why and list what evidence you checked.