voice-agents
Voice agents represent the frontier of AI interaction - humans speaking naturally with AI systems.
- License 许可
- MIT
- 最近更新
- 2026-11-09
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- voice-agents
- risk
- safe
- source
- vibeship-spawner-skills (Apache 2.0)
- date_added
- 1772150400000
- description
- Voice agents represent the frontier of AI interaction - humans speaking naturally with AI systems.
02 资源文件 1
- 参考文档references/detailed-guide.md22.0 KB
03 SKILL.md 正文
Voice Agents
Voice agents represent the frontier of AI interaction - humans speaking naturally with AI systems. The challenge isn't just speech recognition and synthesis, it's achieving natural conversation flow with sub-800ms latency while handling interruptions, background noise, and emotional nuance.
This skill covers two architectures: speech-to-speech (OpenAI Realtime API, lowest latency, most natural) and pipeline (STT→LLM→TTS, more control, easier to debug). Key insight: latency is the constraint. Humans expect responses in 500ms. Every millisecond matters.
84% of organizations are increasing voice AI budgets in 2025. This is the year voice agents go mainstream.
Detailed Guide
Read the detailed guide before executing this skill. It retains the complete procedure and reference material. Treat its safety, prerequisites, and validation requirements as mandatory. For focused work, load the relevant sections; for end-to-end work, read the guide completely.
Production Pipeline Example
""" import { Deepgram } from '@deepgram/sdk'; import { ElevenLabsClient } from 'elevenlabs'; import OpenAI from 'openai';
// Initialize clients const deepgram = new Deepgram(process.env.DEEPGRAM_API_KEY); const elevenlabs = new ElevenLabsClient(); const openai = new OpenAI();
async function processVoiceInput(audioStream) { // 1. Speech-to-Text (Deepgram Nova-3) const transcription = await deepgram.transcription.live({ model: 'nova-3', punctuate: true, endpointing: 300, // ms of silence before end });
transcription.on('transcript', async (data) => { if (data.is_final && data.speech_final) { const userText = data.channel.alternatives[0].transcript; console.log('User:', userText);
// 2. LLM Processing
const completion = await openai.chat.completions.create({
model: 'gpt-4o-mini',
messages: [
{ role: 'system', content: 'You are a concise voice assistant.' },
{ role: 'user', content: userText }
],
max_tokens: 150, // Keep responses short for voice
});
const agentText = completion.choices[0].message.content;
console.log('Agent:', agentText);
// 3. Text-to-Speech (ElevenLabs)
const audioStream = await elevenlabs.textToSpeech.stream({
voice_id: 'voice_id_here',
text: agentText,
model_id: 'eleven_flash_v2_5', // Lowest latency
});
// Stream to user
playAudioStream(audioStream);
}
});
// Pipe audio to transcription audioStream.pipe(transcription); } """
Optimization Tips:
- Start TTS while LLM still generating (streaming)
- Pre-compute first response segment during user speech
- Use Flash/turbo models for latency
Voice Activity Detection Pattern
Detect when user starts/stops speaking
When to use: All voice agents need VAD for turn-taking
VOICE ACTIVITY DETECTION (VAD):
""" VAD Types:
- Energy-based: Simple, fast, noise-sensitive
- Model-based: Silero VAD, more accurate
- Semantic VAD: Understands meaning, best for conversation """
When to Use
- User mentions or implies: voice agent
- User mentions or implies: speech to text
- User mentions or implies: text to speech
- User mentions or implies: whisper
- User mentions or implies: elevenlabs
- User mentions or implies: deepgram
- User mentions or implies: realtime api
- User mentions or implies: voice assistant
- User mentions or implies: voice ai
- User mentions or implies: conversational ai
- User mentions or implies: tts
- User mentions or implies: stt
- User mentions or implies: asr
Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
04 同仓库更多 Skills
dei-assessment
Assess diversity, equity, and inclusion with metrics. TRIGGERS - Use when user needs help with dei-assessment related tasks.
dei-action-plan
Create DEI action plans with goals and initiatives. TRIGGERS - Use when user needs help with dei-action-plan related tasks.
culture-fit-assessment
Design culture add assessment processes. TRIGGERS - Use when user needs help with culture-fit-assessment related tasks.
cowork-to-code-bridge
Use an already-installed, independently verified cowork-to-code bridge to run narrowly approved actions on the user's own macOS, Linux, or WSL2 machine through a local file queue.
counter-offer-response
Draft counter-offer responses for retention. TRIGGERS - Use when user needs help with counter-offer-response related tasks.
continuous-feedback-system
Design continuous feedback systems. TRIGGERS - Use when user needs help with continuous-feedback-system related tasks.