add-middleware
Add a new guardrail or intercept type to the NeMo Relay middleware pipeline
- License 许可
- Apache-2.0
- 所属仓库
- dawsonblock/NEMO
- 最近更新
- 2026-09-30
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- add-middleware
- author
- NVIDIA Corporation and Affiliates
- license
- Apache-2.0
- description
- Add a new guardrail or intercept type to the NeMo Relay middleware pipeline
03 SKILL.md 正文
Add a Middleware Type
Companion Guidance
Use karpathy-guidelines alongside this skill for implementation or review
work. Keep changes scoped, surface assumptions, and define focused validation
before editing.
NeMo Relay supports guardrails (validate/gate) and intercepts (transform) at various pipeline stages. Adding a new middleware type requires changes across all layers.
Use this skill when introducing a new middleware registration surface or adding middleware behavior to a new pipeline stage.
Lock The Design First
Decide these before editing code:
- Is this for tools, LLMs, marks, scope events, or a combination?
- Is it a conditional guardrail, sanitize guardrail, request intercept, or execution intercept?
- Does it run on request input, inner callable execution, stream chunks, or final response output?
- Is the callback fallible, and how should callback failures propagate?
- Does it need both global and scope-local registration?
- What should subscribers and exporters observe in the event payload after this middleware runs?
- If this is an event sanitizer, which of
data,category_profile, andmetadatacan change, and is the event used only as immutable context?
Pipeline Order
Refer to docs/about-nemo-relay/concepts/middleware.mdx for the full diagrams.
- Tool execute: conditional guardrails -> request intercepts -> sanitize request (for events) | execution intercept chain(callable) -> sanitize response
- LLM execute: conditional guardrails -> request intercepts -> sanitize request (for events) | execution intercept chain(callable) -> sanitize response
- Mark and scope events: specialized tool or LLM sanitizer (when applicable) -> mark or scope event sanitizer -> subscriber and exporter dispatch
Tool execution callbacks and each execution-intercept next continuation
return the canonical ToolExecutionResult { result, annotation }. A forwarding
intercept must preserve both fields in ToolExecutionInterceptOutcome; Relay
retains pending_marks separately. Tool sanitize-response guardrails receive
only result. Scope-end event sanitizers govern the annotation after Relay
projects it to category_profile.tool_result_annotation.
Core Steps
- Define or reuse the callback type alias in
crates/core/src/api/runtime/callbacks.rs.
pub type MyNewFn = Box<dyn Fn(&str, Json) -> Json + Send + Sync>;
- Add the registry field to
NemoRelayContextStateincrates/core/src/api/runtime/state.rs.
Add a SortedRegistry<GuardrailEntry<MyNewFn>> or SortedRegistry<Intercept<MyNewFn>>
field to the state struct.
- Add registration and deregistration APIs in
crates/core/src/api/.
Use the existing global_*_registry_api! and scope_*_registry_api! macro
patterns in crates/core/src/api/registry.rs. Both global and scope-local
variants are needed unless the design explicitly rules one out.
- Add chain execution helpers to
NemoRelayContextStateincrates/core/src/api/runtime/state.rs.
Follow the pattern of tool_sanitize_request_chain or tool_request_intercepts_chain.
- Wire the chain into the execute path.
Update the relevant lifecycle owner to call the new chain method at the
appropriate pipeline stage. Tool and LLM paths live in
crates/core/src/api/tool.rs and crates/core/src/api/llm.rs; shared mark and
scope event sanitization lives in crates/core/src/api/shared.rs and is called
from crates/core/src/api/scope.rs.
- Expose the new middleware surface in every affected binding.
Follow the add-binding-feature skill for the cross-binding implementation checklist.
Required Tests
- [ ] Registration and duplicate-name behavior
- [ ] Deregistration and no-op missing-name behavior
- [ ] Ordering by priority
- [ ] Callback failure policy, including fail-open behavior when required
- [ ] Scope-local registration, inheritance, and cleanup on pop
- [ ] Event payload semantics after middleware mutation
- [ ] Tool execution result and annotation preservation, replacement, and removal when the middleware touches tool execution
- [ ] Mark and scope event field semantics, including immutable identity fields
- [ ] Parity coverage in every affected binding
Key References
- Pipeline logic:
crates/core/src/api/tool.rs,crates/core/src/api/llm.rs - Type aliases:
crates/core/src/api/runtime/callbacks.rs - Runtime state and chain builders:
crates/core/src/api/runtime/state.rs - Scope-local registry merging:
crates/core/src/context/registries.rs - Registry:
crates/core/src/registry.rs - Pipeline docs:
docs/about-nemo-relay/concepts/middleware.mdx - Architecture docs:
docs/about-nemo-relay/architecture.mdx - Registration examples:
docs/instrument-applications/advanced-guide.mdx - Validation:
validate-change
04 同仓库更多 Skills
nemo-relay-plugin-observability
Use this skill when choosing or configuring NeMo Relay 0.6 or 0.7 observability through the built-in plugin, subscribers, or exporters, including raw ATOF events, ATIF trajectories, OpenTelemetry, OpenInference, or custom event handling.
nemo-relay-plugin-build
Use this skill when building or packaging reusable NeMo Relay runtime behavior as an embedded configuration component or a manifest-backed `rust_dynamic` native or `worker` gRPC plugin, with deterministic validation and rollback-safe registration.
nemo-relay-plugin-adaptive-tuning
Use this skill when baseline NeMo Relay instrumentation exists and the user wants to configure or evaluate adaptive plugin behavior, including telemetry, state, adaptive_hints, tool_parallelism, ACG, hint consumption, or measured rollout.
nemo-relay-migrate-from-flow
Use this skill when migrating applications, examples, integrations, documentation, manifests, or repository code from NeMo Flow to NeMo Relay across Python, Rust, Node.js, Go, C FFI, CLI, configuration, and observability surfaces.
nemo-relay-integrate-upstream
Use this skill when assessing, extending, or implementing NeMo Relay support in an agent harness or agent framework, including coding agents and orchestration runtimes, when the host lacks Relay support or an existing integration needs deeper coverage. It identifies the host's execution boundaries and extension points, selects an appropriate attachment method for each boundary, and verifies the resulting coverage.
nemo-relay-instrument-typed-wrappers
Use this skill when adding NeMo Relay typed wrappers, domain types, or provider codecs while preserving JSON middleware semantics and caller-visible behavior.