TerminalNexus
I gave my homelab dashboard a voice. Not a push-the-button-and-wait gimmick, an actual hands-free conversation. I'd open a page on my phone, say something, and Alina, the assistant on the box, would answer out loud a moment later, then go quiet and listen for whatever I said next. You could drive it two ways: hold the mic button down like a walkie-talkie and let go when you were done, or flip on hands-free and just talk, no buttons at all. It worked, it ran on my iPhone, and in the end one thing wore me down enough that I stopped using it. Here's how it was built and what I'd change.
A note on where this lives. The dashboard and Alina, the agent behind it, are part of my OpenClaw setup, running in a container on my Linux homelab. So the voice page isn't a standalone toy, it's a front end onto an agent that's already sitting there with its own memory and its own running record of everything we've said.
What I was after
The bar was a real back and forth. I wanted to talk the way you talk to a person on the phone: I say my piece, it answers, I answer that, and nobody is pressing anything. Press-and-hold was the easy version and I built that too, because sometimes you want to own the turn and not have the page guessing. But the one worth the effort was hands-free. That means the page has to listen continuously, work out when I've stopped talking, send what I said, and play the reply, all on a loop, while staying ready to hear me again. That is a full-duplex problem, both directions live at once, and that is the part that makes it interesting.
WebSockets, hand-rolled
The connection underneath it all is a WebSocket. A normal web request is one round trip, you ask, you get an answer, it's over. A WebSocket stays open, and either side can send a message whenever it wants. That two-way always-open channel is what lets the server push audio and status to the browser at the same time the browser is streaming my voice up. Without it you're stuck polling, and polling feels like a CB radio.
I ran it on the plain Node HTTP server the dashboard already used, and rather than pull in a library I implemented the WebSocket frames by hand, to the RFC, parsing and building the frames myself. That sounds like more work than it is, and doing it by hand meant I actually understood every byte going across, including the fragmented frames and the keepalive pings that hold the connection open. The messages themselves are just small JSON envelopes with a type: audio going up, transcription and response and audio coming back, plus status updates so the page can show whether it's listening, thinking, or speaking.
The loop, end to end
A turn goes like this. In hands-free mode the browser captures the mic and watches the volume. When I start talking it records, and when I go quiet for about three seconds it decides I'm done and sends the clip up the socket. In press-and-hold mode there's no guessing, I hold the mic or the spacebar, talk, and the moment I let go the clip goes up. Either way the server hands that audio to a local Whisper model for transcription, passes the text to the voice agent, takes the reply, and runs it through a local text-to-speech model, Kokoro, to get audio back. That audio streams down the same socket and the page plays it automatically, then drops straight back into listening. The whole time, a status line on the page tracks where in that loop we are, so it never feels frozen.
Working out when I'd stopped talking was its own little puzzle, and it has a name: voice activity detection, VAD. The grown-up version is a trained model that knows speech from noise. Mine is cruder. The page reads the raw audio level a few dozen times a second, marks when I cross a volume threshold, and treats a few seconds of quiet as the end of a turn, with a minimum length so a cough doesn't fire it. It reads the level in the time domain rather than the frequency domain, which as a bonus slips past the fingerprint protection some browsers throw at audio. Crude, but it works, and it's all in the browser.
Nothing about a turn is thrown away. The moment one finishes, both what I said and what Alina answered get saved and folded into OpenClaw's memory, indexed through QMD, the vector store it runs. So the conversations don't evaporate the second the page closes, they become something the agent can search by meaning later, the same memory it draws on when I ask it something days afterward.
Both voice models run on the box
The speech-to-text and the text-to-speech are both local by default, running in containers on my own server. My voice goes to my own Whisper, the reply comes back from my own Kokoro, and nothing about the conversation leaves the house. There's a switch in the toolbar that flips the whole pipeline over to OpenAI's hosted voices instead, for when I wanted to hear a voice Kokoro didn't have or just compare the two side by side. But local was the point. The whole appeal was talking to my own machine without renting the privilege.
The face, and the lightning
The page isn't a chat box, it's a face. Alina has a portrait sitting behind the conversation, dim and a little uncanny, staring back while you talk to it. You can swap it: there's a small gallery where you drop in an image, paste one straight from the clipboard, and click a thumbnail to make it the active face. Download a portrait you like, drop it in, and the assistant looks like that from then on.
And the face isn't a still image. A canvas sits over it drawing electric arcs, thin bolts of light that crawl in from the edges toward the center, glow, fade, and respawn somewhere else. The face never moves, but the light and shadow crossing it never stop, so the thing reads as switched on rather than as a screenshot. It costs almost nothing to draw and it's the single cheapest thing I did that made the page feel alive.
The iPhone fight
Getting this working in mobile Safari took some scarring. iOS will not let a web page play audio unless a real tap started it, which is death for a hands-free assistant that wants to talk on its own. The fix is a small trick: on your first tap anywhere, play a single silent sound, which unlocks audio for the rest of the session, and after that the page can speak whenever it likes. I also had to handle touch events alongside mouse events, and add a timeout because iOS sometimes just never tells you a sound finished playing, which would otherwise leave the thing stuck thinking it was still talking.
There's also a pop-out. A button opens the voice chat in its own window, sized to fit a phone screen, so on the desktop it sits in the corner like a little video-call tile and on the phone it's just the face and the conversation with none of the dashboard around it. That pop-out ended up being how I actually used it.
What wore me down
It worked and I still drifted away from it. The reason is latency, and it bites hardest in hands-free mode where you're expecting a natural reply. Every turn is a chain: record, transcribe, let the agent think, synthesize the speech, play it. Each link is quick on its own. Strung together, there's a pause of a few seconds between me finishing and the voice starting, every single time. In a real conversation that gap is small but it's always there, and your brain keeps waiting on it. It's the difference between talking to someone and talking over a satellite link. For typed questions I never noticed. For voice, that delay is the whole experience, and it's the thing I'd throw the most effort at next time, probably by streaming the pieces so the reply starts before all the thinking is finished, instead of waiting for each stage to fully complete.
If you're putting voice on something of your own, the WebSocket and the local models are the easy part. The latency and the mobile audio quirks are where the real work is. Thanks for reading. Happy to get into the weeds in the comments.
Comments