twilio-voice-conversation-relay
Build AI-powered voice agents using Twilio ConversationRelay. Handles real-time speech recognition (ASR), text-to-speech (TTS), and bidirectional audio streaming via WebSocket. Covers TwiML setup, WebSocket message types, LLM integration, streaming responses, and voice provider configuration. Use this skill to build voice bots, IVR replacements, or real-time AI voice assistants on Twilio calls.
- License 许可
- MIT
- 最近更新
- 2026-09-22
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- twilio-voice-conversation-relay
- description
- Build AI-powered voice agents using Twilio ConversationRelay. Handles real-time speech recognition (ASR), text-to-speech (TTS), and bidirectional audio streaming via WebSocket. Covers TwiML setup, WebSocket message types, LLM integration, streaming responses, and voice provider configuration. Use this skill to build voice bots, IVR replacements, or real-time AI voice assistants on Twilio calls.
02 资源文件 1
- 参考文档agents/openai.yaml243 B
03 SKILL.md 正文
Overview
ConversationRelay connects Twilio's telephony layer to your app via a persistent WebSocket. Twilio handles ASR (speech-to-text) and TTS (text-to-speech); your app receives transcripts, calls an LLM, and sends text back for playback.
Caller ←→ Twilio (ASR/TTS) ←→ WebSocket ←→ Your App ←→ LLM
Prerequisites
- Upgraded Twilio account with ConversationRelay access (requires onboarding)
— New to Twilio? See
twilio-account-setup— Start onboarding at: Console > Voice > ConversationRelay — access is not instant - A voice-capable Twilio phone number
TWILIO_ACCOUNT_SIDandTWILIO_AUTH_TOKEN— seetwilio-iam-auth-setup- WebSocket server reachable via
wss://(TLS required) - An LLM integration (OpenAI, Anthropic, etc.)
- For placing calls: see
twilio-voice-outbound-calls
Onboarding: Complete via Console > Voice > ConversationRelay > Onboarding. Select TTS/ASR providers:
- TTS: Deepgram, Amazon Polly, Google Cloud TTS, ElevenLabs
- ASR: Deepgram, Google Cloud STT
Quickstart
Step 1 — Return TwiML pointing to your WebSocket server
Python (Flask)
from flask import Flask
from twilio.twiml.voice_response import VoiceResponse, Connect, ConversationRelay
app = Flask(__name__)
@app.route("/voice", methods=["POST"])
def voice():
response = VoiceResponse()
connect = Connect()
connect.conversation_relay(
url="wss://yourapp.com/ws/voice",
welcome_greeting="Hello! How can I help you today?"
)
response.append(connect)
return str(response)
Node.js (Express)
const { VoiceResponse } = require("twilio").twiml;
app.post("/voice", (req, res) => {
const response = new VoiceResponse();
const connect = response.connect();
connect.conversationRelay({
url: "wss://yourapp.com/ws/voice",
welcomeGreeting: "Hello! How can I help you today?",
});
res.type("text/xml").send(response.toString());
});
Step 2 — Handle WebSocket events and respond with text
Python (websockets)
import asyncio, json, websockets
async def handle_call(websocket):
async for message in websocket:
event = json.loads(message)
if event["type"] == "prompt":
ai_response = await call_llm(event["voicePrompt"])
await websocket.send(json.dumps({"type": "text", "token": ai_response, "last": True}))
async def main():
async with websockets.serve(handle_call, "0.0.0.0", 8080):
await asyncio.Future()
asyncio.run(main())
Node.js (ws)
const WebSocket = require("ws");
const wss = new WebSocket.Server({ port: 8080 });
wss.on("connection", (ws) => {
ws.on("message", async (data) => {
const event = JSON.parse(data);
if (event.type === "prompt") {
const aiResponse = await callLLM(event.voicePrompt);
ws.send(JSON.stringify({ type: "text", token: aiResponse, last: true }));
}
});
});
Security: The
voicePromptfield contains ASR-transcribed caller speech — it is untrusted external input. When passing to an LLM, isolate it as user input within a structured system prompt. Implement topic boundaries and output filtering to prevent the LLM from disclosing system instructions or speaking inappropriate content. ConversationRelay is a pure transport layer with no built-in content safety — any LLM output is spoken to the caller verbatim.
Key Patterns
WebSocket Message Types
Received from Twilio:
| Type | When | Key fields |
|---|---|---|
connected |
WebSocket opened | callSid, streamSid |
prompt |
User finished speaking | voicePrompt (transcript) |
interrupt |
User interrupted TTS | — |
dtmf |
User pressed keypad key | digit |
error |
An error occurred | description |
Sent to Twilio:
| Type | Purpose | Key fields |
|---|---|---|
text |
Send TTS response | token (text), last (bool) |
interrupt |
Stop current TTS | — |
end |
Hang up the call | reason |
Stream LLM Responses Token-by-Token
Lower latency by streaming as the LLM generates output — Twilio starts speaking before the full response is ready.
Python
async for chunk in llm_stream:
await websocket.send(json.dumps({"type": "text", "token": chunk, "last": False}))
await websocket.send(json.dumps({"type": "text", "token": "", "last": True}))
Node.js
for await (const chunk of llmStream) {
ws.send(JSON.stringify({ type: "text", token: chunk, last: false }));
}
ws.send(JSON.stringify({ type: "text", token: "", last: true }));
Voice Configuration
Python
connect.conversation_relay(
url="wss://yourapp.com/ws/voice",
voice="en-US-Neural2-F",
language="en-US",
transcription_provider="deepgram",
speech_model="nova-2-phonecall",
interrupt_by_dtmf=True,
)
Node.js
connect.conversationRelay({
url: "wss://yourapp.com/ws/voice",
voice: "en-US-Neural2-F",
language: "en-US",
transcriptionProvider: "deepgram",
speechModel: "nova-2-phonecall",
interruptByDtmf: true,
});
CANNOT
- No raw audio access — Text in, text out only. For raw audio, use
<Connect><Stream>(Media Streams). - Cannot mix with Media Streams —
<Connect><Stream>and<Connect><ConversationRelay>are mutually exclusive on the same call. No error — one is silently ignored. - No custom STT/TTS engines — Limited to Deepgram + Google (STT) and Deepgram + Amazon Polly + Google + ElevenLabs (TTS).
- No mid-session voice/provider changes — Voice and provider are set at TwiML time. Only
languagecan be switched mid-session via WebSocket message. - No WebSocket auto-reconnection — If the WebSocket drops, the call disconnects. Implement recovery via
<Connect action>URL. - Voice only — No SMS/messaging support. For omnichannel, use Conversation Orchestrator.
- No built-in memory or context — BYO conversation history and context management.
- No LLM integration — Pure transport layer. You bring your own LLM via the WebSocket server.
- No server-side recording via REST API —
record:trueon the Calls API is silently ignored. Must use<Start><Recording>before<Connect>in TwiML. - Not PCI compliant with Voice Intelligence v2 — Do not enable
intelligenceServicein PCI workflows. - ElevenLabs requires account enablement — Accounts without ElevenLabs access get error 64101. Voice IDs (not human-readable names) are required.
- Cannot use ConversationRelay without onboarding — Not available immediately on a new account
- Cannot use non-TLS WebSocket — Server must be reachable via
wss://(TLS required) - Cannot stream audio without
last: true— Twilio won't play audio until it sees alast: truetoken in the stream
Next Steps
- Place outbound calls:
twilio-voice-outbound-calls - Standard IVR without AI:
twilio-voice-twiml
04 同仓库更多 Skills
skill-installer
Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos).
skill-creator
Create or update a Codex skill with appropriately scoped instructions and any needed supporting resources.
review-agent
Perform a read-only, defect-first review of a specified code change and return every actionable finding. Use when another agent delegates review of uncommitted changes, a base-branch diff, a commit, or custom review instructions.
plugin-creator
Create and scaffold plugin directories for Codex with a required `.codex-plugin/plugin.json`, optional plugin folders/files, valid manifest defaults, and personal-marketplace entries by default. Use when Codex needs to create a new personal plugin, add optional plugin structure, generate or update marketplace entries for plugin ordering and availability metadata, or update an existing local plugin during development with the CLI-driven cachebuster and reinstall flow.
openai-docs
Use for Codex models/pricing, scheduled tasks, skills, settings, setup, troubleshooting, customization, automations, and self-knowledge—including 'you,' 'your,' 'this app,' or 'this coding agent' when they refer to Codex—and for OpenAI APIs/products and ChatGPT Work. Also use for model choice/migration, prompting, SDKs, Responses, Realtime, agents, evals, and Chat/Work/Codex comparisons. Do not use for generic app/software tasks that merely mention Codex.
imagegen
Generate or edit raster images when the task benefits from AI-created bitmap visuals such as photos, illustrations, textures, sprites, mockups, or transparent-background cutouts. Use when Codex should create a brand-new image, transform an existing image, or derive visual variants from references, and the output should be a bitmap asset rather than repo-native code or vector. Do not use when the task is better handled by editing existing SVG/vector/code-native assets, extending an established icon or logo system, or building the visual directly in HTML/CSS/canvas.