A local voice assistant, and why the boring option won

I wanted to talk to my house and have it do things: turn off the lights, arm the alarm, ask whether a door was left open. The commercial smart speakers do this, but every word you say goes to a company. I wanted the same convenience with none of the eavesdropping. This is how I built one that runs entirely at home, and the honest ending where the clever solution lost to the simple one. Unfamiliar words are explained on the back-to-basics page.

The plan, and the parts

A voice assistant is really three jobs stitched together:

1. Hearing: turning what you said into text. This is speech-to-text, the same job as dictation.
2. Understanding: working out that "turn off the kitchen light" means a particular switch, and doing it.
3. Speaking: turning the reply back into a voice.

Home Assistant, the popular free home-automation hub, can wire all three together and let you run each part on your own hardware. I ended up spreading the work across two machines: a small, cheap board doing the hearing and speaking, and my main server available for the understanding, when understanding was actually needed.

The hearing and speaking parts were easy

Two small programs on the little board handled voice in and voice out. Home Assistant found them on the network, I picked them in its settings, and the microphone button in its phone app simply started working. This part took an evening and has been rock solid since. If you can follow the dictation guide, you can do this; it is the same idea, pointed at your house instead of a text box.

The interesting part, and the part with the lesson, was the understanding.

Attempt one: run the brain on the little board

My first idea was to run a small language model, the question-answering kind, directly on the cheap board. It technically worked, and it was unusable.

Here is where a number matters, and where those numbers usually go unexplained. The board produced its answer at about 0.4 tokens per second. A token is a small chunk of a word, so tokens per second is just how fast the reply comes out. At 0.4, a short answer like "the kitchen light is off" dribbled out over most of a minute, one fragment at a time, like a fax machine from 1985. Nobody is going to wait that long to hear that a light is off. The board also did not have the spare memory to hold the model comfortably. Verdict: no.

Attempt two: run the brain on the big server

So I moved the understanding to my main server, which has a proper graphics card. The very same kind of model there produced answers at about 165 tokens per second. To put those two numbers side by side: 0.4 was slower than watching someone type with one finger; 165 is faster than you can read along. Night and day.

I did a fair bit of tuning: a short, focused instruction so the model did not waffle, a cap on how long its answers could be, and a setting so that plain commands like "turn off the lights" skipped the model entirely and just did the thing. I also trimmed the list of devices it knew about, because every extra device makes the model a little slower to answer. The result was genuinely good: simple commands instant, spoken questions answered in a couple of seconds.

Attempt three: turn the clever brain off

And here is the uncomfortable finding. For the things people actually say to a house, Home Assistant's own built-in understanding, the plain kind with no language model at all, did everything the clever model did, instantly, with one fewer machine that could break. When the big server is switched off for an update, the voice control keeps working. Nobody in my house asks the light switch philosophical questions.

That third version is the one still running.

What the detour taught me anyway

• Fast is only fast if it is already awake. 165 tokens per second sounds brilliant, but the model took about twelve seconds to wake up from cold, and that wait was the thing people actually noticed and complained about. I ended up keeping it awake on a timer just to hide the delay, a cost the simple version does not have at all.
• Naming beats intelligence. Most of the times the assistant misunderstood me, the fix was renaming a device to what we actually call it, not a cleverer brain. The fancy model was mostly compensating for things being labelled badly.
• Match the tool to the job, not to the excitement. A language model is a remarkable thing. Turning off a light is not a job that needs one.

Would I recommend it

Yes, in the shape it settled into: run the hearing and speaking on any cheap spare board, use Home Assistant's built-in understanding for commands, and keep a local language model in your back pocket for the day you genuinely want it to hold a conversation. Through every version, the thing I cared about held: at no point did anything we said leave the house.

A note on honesty: the little-board part of this I am describing from my own notes rather than a fresh rebuild, and your board's behaviour may differ. The hearing and speaking layer, though, has been the single most dependable part of the whole experiment.