agentic-optimization

Agentic Optimization

This skill analyzes AI agent telemetry traces from Application Insights using the aira.exe CLI tool. It performs anomaly detection, trend analysis, and generates performance statistics to help identify issues in AI agent behavior.

Prerequisites

• The user must be logged in to Azure CLI (az login)
• The user must provide Application Insights resource details: Subscription ID, Resource Group, and Component Name

Steps

1–2. Check investigation notes and gather inputs

Follow the Standard Skill Preamble to check for existing investigation context and gather inputs.

In addition to the standard App ID, this skill requires the following Application Insights identity fields:

• Subscription ID: Azure subscription GUID
• Resource Group: Resource group containing the Application Insights resource
• Component Name: The Application Insights component name (resource name)

These are typically available in investigation-notes.md. If only a Resource ID is available, parse the subscription ID, resource group, and component name from the ARM resource ID format:

/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/microsoft.insights/components/{componentName}

If none of the above are available, follow Identify Application Insights Resource to locate the resource.

Optional inputs — ask the user if they want to narrow the analysis:

• Agent name: Filter by a specific agent name
• Agent version: Filter by a specific agent version
• Limit: Maximum number of records to analyze (1–50,000). Defaults to 1000 — suitable for initial exploration; increase for thorough analysis.
• Time range: Start and end times for the analysis window (ISO 8601 UTC). Defaults to the last 24 hours. Widen if the user wants historical analysis.

2b. Create or update investigation notes

If investigation-notes.md does not already exist in the working directory, create it now with the resource context gathered in steps 1–2. This ensures downstream skills (especially deep-analysis) can pick up the context without re-asking the user.

Write the following fields to investigation-notes.md using the format defined in investigation-notes.md:

• Subscription ID
• Resource Group
• Component Name
• Resource ID (constructed from the above: /subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/microsoft.insights/components/{componentName})

If the file already exists, verify the resource context matches and update if needed.

3. Run the analyze command and present results

Run the script in run-analyze.md to execute the aira.exe analyze command. The script acquires a fresh access token, invokes the CLI with JSON output, and post-processes the results into a readable summary with operation IDs extracted — all in a single run.

Access token: This skill uses the Application Insights data-plane token (https://api.applicationinsights.io), which is different from the profiler dataplane token used by other skills. The script handles token acquisition automatically.

Execution time: The analysis may take a while depending on the volume of telemetry data. Inform the user that this may take some time, especially with large --limit values or wide time ranges.

The script output contains everything needed to present results and offer deep-dive. Present it to the user with the following interpretation:

1. Lead with the summary header — State total records, agent count, and time range.

2. Highlight per-agent performance — Call out:

   • Agents with P95 > 5,000ms as high-latency
   • Large gaps between mean and P95 (indicates inconsistent performance / tail latency)
   • Low token counts with high latency (latency is likely in tool calls, not LLM)

3. Explain the operation breakdown — The script shows per-operation-type stats:

   • invoke_agent — overall agent invocation (parent span)
   • chat — LLM model calls (check latency and token counts)
   • execute_tool — tool/API calls (often the bottleneck; check byTool breakdown)

4. Interpret trends — Notable trends (confidence ≥ 0.5) are shown. Explain:

   • Duration increasing → potential regression or growing prompt size
   • Token usage increasing → context window growth, possible cost concern
   • Duration decreasing → improvement or reduced workload

5. Present anomaly operations for deep-dive — The script extracts operation IDs from anomaly spikes and displays them in a numbered table. Present this table directly and ask the user which operation they'd like to deep-dive into. Each row includes:

   • Operation ID (needed for deep-analysis skill handoff)
   • Duration and severity
   • Span types and model for context

Raw JSON: The script saves the full JSON to aira-output.json in the working directory. Let the user know they can inspect this file for the complete data.

4. Deep-dive handoff

If the anomaly operations table has entries, write the key findings to investigation-notes.md before handing off:

1. Add a "Key Findings" section with:

   • Agent name, time window, and summary statistics (mean, P95, max latency)
   • The bottleneck layer (LLM vs tool calls) and which tools are slowest
   • Trend direction if notable

2. Add an "Anomaly Operations" section with the top anomaly operation IDs, durations, and severity scores

Then ask the user which operation to investigate further:

• Cross-resource deep analysis (recommended): Hand off the selected operation ID to the deep-analysis skill to trace the operation across downstream services (tools, APIs, databases) and see where time was spent. The operation ID and resource context will be available in investigation-notes.md.
• Response context: Run aira.exe response-context --response-id <responseId> to see the full agent conversation flow for that operation. Important: use the responseId field (not the operationId) as the --response-id argument — these are different identifiers. The anomaly table output includes both.
• Compare agent versions: If the issue correlates with a specific agent version, suggest aira.exe compare-versions.

If the anomaly operations table is empty (no spikes detected), suggest:

• Widen the time range to capture more data and potential anomalies
• Increase the limit for more thorough analysis
• Re-run with a specific agent name to focus the analysis
• Check the raw JSON (aira-output.json) for lower-severity anomalies that didn't meet the threshold

5. Additional follow-up options

Beyond the deep-dive, offer these follow-up options based on findings:

• Narrow the analysis: Re-run with --agent-name and/or --agent-version filters to focus on a specific agent
• Adjust the time window: Narrow or widen the analysis period to isolate issues
• Cross-reference with profiler data: If latency issues are found in downstream services, suggest using the perf-optimization skill to correlate with profiler traces

Tips

• Always confirm the Application Insights resource with the user before running the analysis.
• Start with a default analysis (no filters, last 24 hours, limit 1000) to get a broad overview, then narrow down based on findings.
• The script defaults to JSON output (-o json) with post-processing that extracts a readable summary and operation IDs in one pass. The raw JSON is saved to aira-output.json for user inspection.
• The --limit parameter caps the number of telemetry records analyzed. The CLI defaults to 1000. For thorough analysis, increase up to 50,000.
• The script defaults to a 24-hour time window. Widen it for historical analysis; narrow it to isolate recent incidents.
• When comparing agent performance across versions, note the agent name from the results and suggest the user run a follow-up comparison.
• If the CLI returns exit code 1 with "No telemetry records found", check the error message — it now provides actionable guidance on what to try next.
• The goal is to reach "ready for deep-dive" in a single run — present summary + operation IDs together so the user can pick one immediately.

References

For the investigation notes format and read/write protocol, see:

• Investigation Notes

For finding the Application Insights resource, see:

• Identify Application Insights Resource

For the full CLI specification (all available commands), see:

• CLI Specification