AGENT SKILL

inference-service

Use when designing, reviewing, or debugging TypeScript/JavaScript inference backends involving request admission, queues, engine batching, SSE streaming, cancellation, caching, fallbacks, circuit breakers, or GPU capacity.

License 许可
MIT
最近更新
2026-08-20
Stars
★ 0
Forks
0

01 前言 Front Matter

name
inference-service
description
Use when designing, reviewing, or debugging TypeScript/JavaScript inference backends involving request admission, queues, engine batching, SSE streaming, cancellation, caching, fallbacks, circuit breakers, or GPU capacity.

03 SKILL.md 正文

Inference Service

<HARD-GATE> There is no universal inference topology. Before choosing components, define the latency SLO, quality contract, delivery semantics, tenant isolation, privacy/retention rules, cancellation behavior, and measured load. Never trade correctness or isolation for an assumed availability improvement. </HARD-GATE>

Contract First

Record:

  • synchronous response versus accepted asynchronous job;
  • maximum prompt/output size, concurrency, and rate limits per tenant;
  • model/version/policy pinning and whether degradation is allowed;
  • at-most-once, at-least-once, or best-effort processing;
  • retry/idempotency contract and retention/deletion requirements;
  • latency to first token, total latency, throughput, and cost budgets.

Do not promise zero loss. Durable brokers still require replicated storage, producer acknowledgement, consumer acknowledgement, idempotency, replay policy, dead-letter handling, and tested disaster recovery.

Select Architecture from Evidence

  • A direct service-to-engine call can be correct at low measured load when the request remains synchronous and bounded.
  • A queue is optional: add one for durable accepted jobs, admission smoothing, or explicit backpressure—not by habit. Specify when a request becomes accepted and what cancellation means after enqueue.
  • Keep GPU/model runtimes isolated from the Node.js control plane when their failure, memory, or scaling characteristics differ.
  • Prefer engine batching when supported, but tune from throughput/latency measurements. Client batching is justified only by a documented engine or transport limitation.

Admission and Streaming

Authenticate and authorize before enqueue; validate model, tenant, token budget, and request schema. Apply per-tenant quotas and idempotency keys.

For SSE, propagate an AbortSignal, bound buffers and output tokens, implement backpressure, heartbeat only when infrastructure requires it, and handle disconnect cleanup idempotently. Do not acknowledge durable work merely because the client disconnected; follow the declared delivery contract.

Cache, Breaker, and Fallback Safety

A semantic cache is opt-in, not a default. Disable it for sensitive data, rapidly changing policy, or outputs where semantic similarity cannot preserve correctness. If permitted, partition and key by tenant, authorization scope, model, version, policy, parameters, and safety configuration; define TTL, invalidation, encryption, and auditability.

Use a circuit breaker around a named failing dependency with measured thresholds, minimum sample size, cooldown, and half-open probes. Preserve accepted-work semantics when it opens.

Use a fallback only when the quality contract explicitly permits the alternate model, stale/cache age, or asynchronous response. Label degraded output and meter it. Never silently substitute another tenant's cache entry, model/version, or lower-quality answer.

Observability and Verification

Measure admission rejects, queue depth/age, time to first token, total latency, tokens/second, cancellation latency, retry/replay counts, cache hits by tenant/model/policy, breaker state, fallback rate, GPU utilization/memory, OOMs, and cost.

Test overload, duplicate delivery, process death before/after acknowledgement, broker outage, partial SSE disconnect, tenant isolation, cache invalidation, model rollout, breaker recovery, fallback prohibition, and graceful shutdown. Use fault injection before claiming durability or degradation behavior.

04 同仓库更多 Skills

MinhDuyDEV
MinhDuyDEV/md-pikit

writing-skills

Use when creating new skills, editing existing skills, or verifying skills work before deployment - applies TDD to process documentation by testing with subagents before writing, iterating until bulletproof against rationalization. Includes complete pressure testing methodology.

★ 0
MinhDuyDEV
MinhDuyDEV/md-pikit

verification-before-completion

Use when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always

★ 0
MinhDuyDEV
MinhDuyDEV/md-pikit

typescript-coding-standards

Use when writing, refactoring, or reviewing TypeScript code that needs strong domain modeling, typed errors, schema parsing, safe adapters, test seams, or maintainable module boundaries.

★ 0
MinhDuyDEV
MinhDuyDEV/md-pikit

test-driven-development

Use when implementing a feature, bugfix, or behavior-changing refactor — require a failing behavior test before production code, then make the smallest change that passes.

★ 0
MinhDuyDEV
MinhDuyDEV/md-pikit

source-driven-development

Use when unfamiliar libraries, dependency internals, external APIs, framework behavior, migrations, or current ecosystem guidance require official documentation, version-matched source, read the tests, and cited evidence.

★ 0
MinhDuyDEV
MinhDuyDEV/md-pikit

security-and-hardening

Use when auditing security, implementing authentication or authorization, handling secrets, or validating data across external trust boundaries.

★ 0