AGENT SKILL

ams-voice-ai-pro

暂无简介

License 许可
MIT
所属仓库
TanNhatCMS/jarvis
最近更新
2026-11-07
Stars
★ 0
Forks
1

03 SKILL.md 正文

Voice AI Pro

A builder's guide to speech technology: speech-to-text (STT), text-to-speech (TTS), real-time voice agents, and telephony. Covers the pipeline architecture, latency budgets, and the UX details that separate usable voice products from demos.

Overview

A voice pipeline is a chain: capture → voice activity detection → STT → intelligence (LLM/dialog) → TTS → playback. End-to-end latency is the product metric that matters most — every 500ms of delay makes conversation feel broken. The architecture trick is streaming and overlap: start transcribing before the user finishes, start synthesizing before the LLM finishes, play audio while generating the rest.

When to use

  • Transcribing meetings, calls, or audio files (batch or streaming).
  • Adding TTS voiceovers to content or apps.
  • Building conversational voice agents (support lines, IVR replacement).
  • Wake-word / push-to-talk interfaces.
  • Speaker diarization ("who said what") for multi-speaker audio.

Core concepts

  • STT (speech-to-text). Batch models for files (highest accuracy); streaming models for live audio (partial results, lower latency). Word-level timestamps enable captions and search.
  • VAD (voice activity detection). Detects speech vs silence/noise. Drives endpointing (when the user is "done") and saves compute by skipping silence.
  • Endpointing. Deciding the user finished speaking. Too aggressive = interruptions; too lax = awkward pauses. Tune silence thresholds per use case (300–800ms typical).
  • TTS. Neural voices, near-human quality. Control via SSML (pauses, emphasis, pronunciation) or plain text. Voice cloning exists — treat as consent-gated.
  • Barge-in. Letting the user interrupt the agent mid-speech. Requires echo cancellation and fast pipeline teardown; without it, voice agents feel like bad IVR.
  • Diarization. Assigning transcript segments to speakers. Accuracy drops with overlapping speech and similar voices.
  • Telephony. SIP trunks / CPaaS providers bridge phone calls to your pipeline (typically via WebSocket media streams, 8kHz μ-law audio).
  • Duplex architecture. Full-duplex (user and agent can speak simultaneously, like a phone call) vs half-duplex (walkie-talkie turns). Full-duplex needs barge-in + echo cancellation.

Practical workflow

1. Choose the pipeline shape.

  • Transcription feature → batch STT API, store transcript + timestamps.
  • Live captions → streaming STT with partial results.
  • Voice agent → streaming STT + LLM + streaming TTS, all overlapped.

2. Set the latency budget. Work backwards from a target (e.g., <1.2s user-stops-speaking to agent-starts-speaking): VAD/endpointing ~300ms, STT final ~200ms, LLM first token ~400ms, TTS first audio ~300ms. Measure each stage separately.

3. Stream everything.

mic → VAD → streaming STT (partials) → on endpoint: final transcript
final transcript → LLM (stream tokens) → sentence buffer → TTS (stream audio) → playback

Start TTS on the first complete sentence, not the full response. Play while generating.

4. Handle the messy middle. Partial transcripts will revise themselves — display them as provisional. Handle filler ("um"), restarts, and background noise with VAD tuning and noise suppression.

5. Telephony integration. Receive call via SIP/WebSocket, decode μ-law to PCM, run the pipeline, encode back. Test with real phone audio (8kHz, compressed) — lab-mic quality lies.

6. Consent and disclosure. Inform callers they're speaking to an AI; get explicit consent for recording where law requires (two-party consent jurisdictions).

Common pitfalls

  • Latency death by sequential design. Waiting for full STT → full LLM → full TTS before playing anything yields 4–6s pauses. Overlap stages.
  • No barge-in. Users will talk over the agent; without interruption handling they must wait through every prompt.
  • Endpointing mistuned. Cutting users off mid-thought (too fast) or long dead air (too slow) — tune per population, and use semantic endpointing (LLM judges completeness) for complex queries.
  • Echo/feedback. Speaker audio re-entering the mic creates loops. Use echo cancellation (AEC) on devices, headsets in testing.
  • Voice cloning without consent. Cloning a real person's voice — especially without explicit permission — is legally radioactive and banned by most TTS providers' terms. Use licensed/stock voices.
  • Ignoring audio quality. 8kHz phone audio, car noise, accents — test on the actual channel, not clean studio recordings. Accuracy numbers from benchmarks won't match your users.
  • No fallback for STT failures. Silence, unintelligible audio, or API errors need graceful handling ("I didn't catch that — could you repeat?") not crashes.
  • Cost per minute. Streaming STT + LLM + TTS compounds; a 10-minute call can cost dollars. Monitor, set session caps, and consider smaller models for the dialog layer.

04 同仓库更多 Skills

TanNhatCMS
TanNhatCMS/jarvis

web-scrape

Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open/fetch fails on JS-heavy pages, or when you need clean markdown/structured extraction from a site. Requires FIRECRAWL_API_KEY.

★ 0
TanNhatCMS
TanNhatCMS/jarvis

design-taste-frontend

tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML/CSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, "AI-made", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.

★ 0
TanNhatCMS
TanNhatCMS/jarvis

test-driven-development

Use when implementing any feature or bugfix, before writing implementation code

★ 0
TanNhatCMS
TanNhatCMS/jarvis

systematic-debugging

Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes

★ 0
TanNhatCMS
TanNhatCMS/jarvis

requesting-code-review

Use when completing tasks, implementing major features, or before merging to verify work meets requirements

★ 0
TanNhatCMS
TanNhatCMS/jarvis

brainstorming

You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).

★ 0