local-voice-agent
暂无简介
- License 许可
- Apache-2.0
- 最近更新
- 2026-10-24
- Stars
- ★ 0
- Forks
- 0
02 资源文件 1
- 参考文档skill.json617 B
03 SKILL.md 正文
local-voice-agent
Local Voice Agent
Build a voice conversation system: mic → local STT → agent brain → local TTS → speakers. Half-duplex (mic gated while agent speaks). Private by default — run the models locally.
Architecture (proven)
mic → Ears (whisper.cpp :2022 /v1/audio/transcriptions)
→ Brain (warm persistent agent session, streaming)
→ Sentence chunker
→ Mouth (Kokoro :8880 /v1/audio/speech → PCM 24k int16)
→ speakers
State machine: keydown→listening → keyup→thinking → first audio→speaking → done→idle. While speaking: mic closed, transcription off, no self-hear. Press talk while speaking → interrupt + reopen mic. Typed input races PTT via asyncio.wait(FIRST_COMPLETED).
Gotchas (real, cost time — read references/)
- whisper.cpp server uses
--port(NOT-p); default 8080; serves ONLY/inference, not/v1/audio/transcriptions. Put a thin adapter in front to expose the OpenAI route. Seereferences/whisper_kokoro.md. - Kokoro PyPI
kokoro[fastapi]extra does NOT ship a server module in current versions. Write a small FastAPI wrappingKPipeline(returns raw PCM). Seereferences/whisper_kokoro.md. - Brain = Hermes locally:
hermes serveWS gateway (:9119) needs dashboard cookie auth (401 headless). The proven headless path ishermes chat -q "<text>" -Q --resume <id>; session_id prints to STDERR. Capture it, reuse it → warm persistent session. Seereferences/hermes_brain.md. - Audio I/O needs a real machine: sounddevice needs PortAudio; pynput needs an X display. Headless servers can't capture/play audio — run the loop on the desk machine.
Workflow rule (user correction, 2026-07-15)
Before building a visual/conversational system from a spec, ASK FIRST. Propose the concept (or state→visual mapping) and wait for approval before writing code. The user explicitly required this twice: "Proceed, and always ask like you just did before any build." For the visualizer he picked a concept, then I proposed how each of the 5 states maps to the scene and waited. Embed this in any creative/build task for this user.
Companion: browser visualizer (reads the same voice bus)
A fullscreen Canvas visualizer can react to the voice agent via the ~/voice-line/
status files. Server is stdlib-only, read-only, binds 127.0.0.1:8777; /state returns
{state, level, alert}. Critical stomp-tolerance: if .voice_waveform is <2s fresh,
report speaking regardless of .voice_state. See references/voice_bus_visualizer.md.
Interruptibility & signal bus (root-cause fixes)
- Blocking playback kills interrupts.
sd.OutputStream.write(whole_sentence)blocks the event loop, sostop_playback()from another task can't run mid-sentence. Fix: play inloop.run_in_executor, write in small chunks (e.g. 2048 samples), check athreading.Eventstop flag between chunks (~40ms cancel latency). Root cause = single blocking call, not the interrupt logic. - Feed the visualizer REAL level. Don't write
set_waveform([0.0]*64). Downsample the actually-played PCM to 64 abs-normalized floats and push it to.voice_waveformso the visualizer's speaking animation tracks the voice.
Remote / thin-client deployment (audio ≠ brain machine)
When the agent CLI (hermes) lives on a server but mic/speakers are on a laptop, run the
audio loop on the laptop and reach the rest over LAN. Make endpoints env-driven:
HERMES_HOST(default 127.0.0.1) → Whisper/Kokoro URLshttp://$HERMES_HOST:2022|8880.HERMES_BIN(default "hermes") →.split()it soHERMES_BIN="ssh user@host hermes"runs the brain remotely. SSH argv trap:create_subprocess_exec(["ssh","host","hermes","chat","-q",text,"-Q"])passes each token separately; the remote sshd re-joins them and the remote shell re-splits on spaces, so the quotes aroundtextare LOST →hermes: error: unrecognized arguments(a multi-word query splits into stray args). Fix: when prefix[0]=="ssh", build ONE remote string with the query shell-quoted:remote = "hermes chat -q " + shlex.quote(text) + " -Q ..."thencmd = [*ssh_prefix, remote]. Local (non-ssh) path keeps the argv list unchanged.- BIND 0.0.0.0, NOT 127.0.0.1. Whisper adapter (:2022), Kokoro (:8880), and the
visualizer server (:8777) all default to
127.0.0.1— unreachable from the laptop. For thin-client, bind0.0.0.0on every service the laptop hits (uvicornhost=,ThreadingHTTPServer(("0.0.0.0", PORT))). Restart the actual PROCESS after the edit; a running daemon keeps its old bind. Identify by port, kill that PID, relaunch. - run.sh guard: don't gate on
if [ "$HERMES_HOST" = 127.0.0.1 ]— the env var often fails to propagate into the script and it wrongly boots local torch/uvicorn. PROBE the target instead:if curl -s --max-time 2 http://$HERMES_HOST:2022/health; then skip local servers; else start them; fi. Probing the real endpoint is robust to env drift. - Needs keyless SSH (ed25519 → server
authorized_keys). If server user has NO password, key auth is the ONLY way in;ssh-copy-idcan't run, so append the laptop'sid_ed25519.pubto the server's~/.ssh/authorized_keysdirectly. Seereferences/remote_client.md.
Pitfalls that crashed a real session
- Duck/context-manager
__exit__reads an attr__init__never set.Duck.__exit__readself._vol, but__enter__only set it when Spotify was running →AttributeErrorkilled the turn when Spotify was absent. Rule: init EVERY attribute referenced in__exit__/cleanup in__init__(self._vol = None), so partial-enter never crashes exit. - Scope: "voice agent" ≠ "browser app". User expected to open the voice agent in a
browser like a dashboard. The talk loop is a LOCAL python app (mic/speakers), not a web
UI — only the visualizer is browser-served. State this split upfront: visualizer = browser
(
http://<host>:8777); talking = local app. Browser-mic voice is a separate, unbuilt feature.
Remote brain via OpenLive + ACP over SSH
When the user runs OpenLive (a desktop voice app) on their laptop and wants Hermes on a
headless host to be the brain, OpenLive drives Hermes via ACP over keyless SSH. See
references/openlive_acp_ssh.md for the full setup: install hermes-agent[acp] in the
Hermes venv, generate an ed25519 key, add it to the laptop's authorized_keys, and point
OpenLive's Hermes agent at a project path on the host.
Files shape
main.py (orchestrator) · ears.py (STT) · brain.py (agent) · mouth.py (TTS) · ptt.py (hotkey + held-state) · ducking.py (optional Spotify) · signals.py (status bus) · run.sh (start servers). Keep it to the files the spec needs; no extra layers.
Verify before claiming done
Whisper health 200 · Kokoro returns PCM · brain returns a reply + reuses session_id · full text loop (transcript→brain→PCM) works. Real mic/speaker only testable on desk machine.