The first version was not bad. We built an AI agent that answers a phone line and books a vehicle service appointment on its own: Vonage brings in the call, audio streams over a WebSocket to a Node and Express service, Google speech-to-text turns it into a transcript, an OpenAI agent decides what to say, Google text-to-speech turns the answer back into audio, and the caller hears it. It worked. Customers booked appointments at two in the morning without anyone being awake.
The transcription piece inside it, though, was wrong in a way that working software can be wrong. It was welded to that one product. Every assumption about session handling, noise, and stream lifecycle lived inside the agent, and the only way to get transcription into a second product was to copy it and start diverging.
What the coupling actually cost
Three things showed up before scale did. Handling several WebSocket clients at once was awkward, because connection management was written for one caller at a time. Noise cancellation was tangled with call logic, so tuning it meant touching the agent. And streaming stability was hard to reason about, because the thing that owned the stream also owned the conversation.
None of that is a bug. Each one is the same structural problem seen from a different angle: a capability the business would clearly want again was shaped like a feature of one app.
Making it a service rather than a library
A library would have solved reuse and nothing else. What we built instead is a standalone microservice: you point audio at it, you get transcripts back, and you never learn how streaming speech-to-text works. That distinction drove every decision after it.
Clients authenticate with an API key and secret, and get a JWT that authorises the session. Not because a portfolio of internal apps needs hard security theatre, but because the moment transcription is a service, somebody outside your team is going to integrate it, and you want that boundary to exist from day one rather than be retrofitted.
The abstraction over Google streaming speech-to-text is deliberately thin: connect, close, writeToStream, stopStream, and events for stt_started, transcript and user_speaking. Thin abstractions age well. Thick ones become a second product you did not intend to maintain.
The queue was the part I would defend hardest
Concurrent load on a real-time audio service does not degrade gracefully on its own. When capacity runs out, the honest options are to refuse the connection or to hold it. We hold it: connections queue, allocation is fair, the client is told it is queued, and it connects automatically when resources free up.
A queue that tells the client it is queued is a feature. A queue that quietly drops it is an outage with better manners.
That single behaviour is the difference between a service an integrator trusts and one they wrap in retry logic and complain about.
Two engines on purpose
Google speech-to-text runs in production because it is reliable and accurate enough for live calls. We also keep Faster-Whisper, a fine-tuned Whisper, and IBM Granite wired into the same modular socket handler for internal testing and benchmarking.
Keeping a second path alive costs something, and it is worth it. A provider you cannot benchmark is a provider you cannot argue with. When accuracy drops in a noisy service bay, I want a number comparing engines on our own audio, not a support ticket.
The rule I took from it
If a capability gets asked for twice, it is a service, and the second ask is late. The work that made the difference here was not the model choice or the latency tuning. It was noticing that transcription was infrastructure while it was still disguised as a feature, and moving it before a third product needed it.
A React developer portal now handles sign-up, applications, keys, docs and usage stats. Any new project gets transcription in an afternoon. Next on it: an analytics dashboard for quality and latency, more languages, and quality-of-service handling so premium users sit at the front of the queue.