AGENT SKILL

add-middleware

Add a new guardrail or intercept type to the NeMo Relay middleware pipeline

License 许可
Apache-2.0
所属仓库
dawsonblock/NEMO
最近更新
2026-09-30
Stars
★ 0
Forks
0

01 前言 Front Matter

name
add-middleware
author
NVIDIA Corporation and Affiliates
license
Apache-2.0
description
Add a new guardrail or intercept type to the NeMo Relay middleware pipeline

03 SKILL.md 正文

Add a Middleware Type

Companion Guidance

Use karpathy-guidelines alongside this skill for implementation or review work. Keep changes scoped, surface assumptions, and define focused validation before editing.

NeMo Relay supports guardrails (validate/gate) and intercepts (transform) at various pipeline stages. Adding a new middleware type requires changes across all layers.

Use this skill when introducing a new middleware registration surface or adding middleware behavior to a new pipeline stage.

Lock The Design First

Decide these before editing code:

  • Is this for tools, LLMs, marks, scope events, or a combination?
  • Is it a conditional guardrail, sanitize guardrail, request intercept, or execution intercept?
  • Does it run on request input, inner callable execution, stream chunks, or final response output?
  • Is the callback fallible, and how should callback failures propagate?
  • Does it need both global and scope-local registration?
  • What should subscribers and exporters observe in the event payload after this middleware runs?
  • If this is an event sanitizer, which of data, category_profile, and metadata can change, and is the event used only as immutable context?

Pipeline Order

Refer to docs/about-nemo-relay/concepts/middleware.mdx for the full diagrams.

  • Tool execute: conditional guardrails -> request intercepts -> sanitize request (for events) | execution intercept chain(callable) -> sanitize response
  • LLM execute: conditional guardrails -> request intercepts -> sanitize request (for events) | execution intercept chain(callable) -> sanitize response
  • Mark and scope events: specialized tool or LLM sanitizer (when applicable) -> mark or scope event sanitizer -> subscriber and exporter dispatch

Tool execution callbacks and each execution-intercept next continuation return the canonical ToolExecutionResult { result, annotation }. A forwarding intercept must preserve both fields in ToolExecutionInterceptOutcome; Relay retains pending_marks separately. Tool sanitize-response guardrails receive only result. Scope-end event sanitizers govern the annotation after Relay projects it to category_profile.tool_result_annotation.

Core Steps

  1. Define or reuse the callback type alias in crates/core/src/api/runtime/callbacks.rs.
pub type MyNewFn = Box<dyn Fn(&str, Json) -> Json + Send + Sync>;
  1. Add the registry field to NemoRelayContextState in crates/core/src/api/runtime/state.rs.

Add a SortedRegistry<GuardrailEntry<MyNewFn>> or SortedRegistry<Intercept<MyNewFn>> field to the state struct.

  1. Add registration and deregistration APIs in crates/core/src/api/.

Use the existing global_*_registry_api! and scope_*_registry_api! macro patterns in crates/core/src/api/registry.rs. Both global and scope-local variants are needed unless the design explicitly rules one out.

  1. Add chain execution helpers to NemoRelayContextState in crates/core/src/api/runtime/state.rs.

Follow the pattern of tool_sanitize_request_chain or tool_request_intercepts_chain.

  1. Wire the chain into the execute path.

Update the relevant lifecycle owner to call the new chain method at the appropriate pipeline stage. Tool and LLM paths live in crates/core/src/api/tool.rs and crates/core/src/api/llm.rs; shared mark and scope event sanitization lives in crates/core/src/api/shared.rs and is called from crates/core/src/api/scope.rs.

  1. Expose the new middleware surface in every affected binding.

Follow the add-binding-feature skill for the cross-binding implementation checklist.

Required Tests

  • [ ] Registration and duplicate-name behavior
  • [ ] Deregistration and no-op missing-name behavior
  • [ ] Ordering by priority
  • [ ] Callback failure policy, including fail-open behavior when required
  • [ ] Scope-local registration, inheritance, and cleanup on pop
  • [ ] Event payload semantics after middleware mutation
  • [ ] Tool execution result and annotation preservation, replacement, and removal when the middleware touches tool execution
  • [ ] Mark and scope event field semantics, including immutable identity fields
  • [ ] Parity coverage in every affected binding

Key References

  • Pipeline logic: crates/core/src/api/tool.rs, crates/core/src/api/llm.rs
  • Type aliases: crates/core/src/api/runtime/callbacks.rs
  • Runtime state and chain builders: crates/core/src/api/runtime/state.rs
  • Scope-local registry merging: crates/core/src/context/registries.rs
  • Registry: crates/core/src/registry.rs
  • Pipeline docs: docs/about-nemo-relay/concepts/middleware.mdx
  • Architecture docs: docs/about-nemo-relay/architecture.mdx
  • Registration examples: docs/instrument-applications/advanced-guide.mdx
  • Validation: validate-change

04 同仓库更多 Skills

dawsonblock
dawsonblock/NEMO

nemo-relay-plugin-observability

Use this skill when choosing or configuring NeMo Relay 0.6 or 0.7 observability through the built-in plugin, subscribers, or exporters, including raw ATOF events, ATIF trajectories, OpenTelemetry, OpenInference, or custom event handling.

★ 0
dawsonblock
dawsonblock/NEMO

nemo-relay-plugin-build

Use this skill when building or packaging reusable NeMo Relay runtime behavior as an embedded configuration component or a manifest-backed `rust_dynamic` native or `worker` gRPC plugin, with deterministic validation and rollback-safe registration.

★ 0
dawsonblock
dawsonblock/NEMO

nemo-relay-plugin-adaptive-tuning

Use this skill when baseline NeMo Relay instrumentation exists and the user wants to configure or evaluate adaptive plugin behavior, including telemetry, state, adaptive_hints, tool_parallelism, ACG, hint consumption, or measured rollout.

★ 0
dawsonblock
dawsonblock/NEMO

nemo-relay-migrate-from-flow

Use this skill when migrating applications, examples, integrations, documentation, manifests, or repository code from NeMo Flow to NeMo Relay across Python, Rust, Node.js, Go, C FFI, CLI, configuration, and observability surfaces.

★ 0
dawsonblock
dawsonblock/NEMO

nemo-relay-integrate-upstream

Use this skill when assessing, extending, or implementing NeMo Relay support in an agent harness or agent framework, including coding agents and orchestration runtimes, when the host lacks Relay support or an existing integration needs deeper coverage. It identifies the host's execution boundaries and extension points, selects an appropriate attachment method for each boundary, and verifies the resulting coverage.

★ 0
dawsonblock
dawsonblock/NEMO

nemo-relay-instrument-typed-wrappers

Use this skill when adding NeMo Relay typed wrappers, domain types, or provider codecs while preserving JSON middleware semantics and caller-visible behavior.

★ 0