The App Interrupted Itself and Repeated the Same Answer

In your realtime pipeline, how far do you trust the processing the hardware does for you?

I had echo cancellation on and considered the matter closed. It was on, it was working, and the app still cut itself off mid-sentence.

The report

Three conversational voice apps share the same audio code. They capture the mic at 24 kHz, stream it to a realtime API, and play back what comes down. The server detects speech onset and handles turn-taking, so the client just keeps the stream flowing.

The report read: "still see the pattern where it interrupts and repeats sometimes during a conversation."

"Sometimes" describes the whole nature of the bug. Always means the wiring is wrong. Sometimes means you're near a threshold.

The app was hearing its own voice

Echo cancellation was on. There's an explicit call requesting hardware acoustic echo cancellation on the audio engine, and it works.

The problem is that it isn't perfect.

Echo cancellation removes 20–35 dB relative to the source. With a loud speaker volume or on certain routes, the residual occasionally crosses the server's speech-detection threshold. Then:

The server decides "the user started speaking."
It cancels the response it was generating. → the app interrupts itself.
It treats that echo as user speech and answers again. → it repeats itself.

The two symptoms in the report were the same cause seen twice.

Looking at the code, there was exactly one layer of defence. Every mic frame went to the server unconditionally. There was no software gate at all.

A second problem surfaced with it. When a user genuinely interrupts, the server cancels generation — but the seconds of audio already sitting in the client's playback queue keep playing. The app talks over the user.

Where would you put the threshold?

There's a fork here. Raising the server-side speech-detection threshold is the easiest option. One line.

I didn't use it. It's a blunt instrument that also stops hearing users who speak quietly. Blocking real users to block echo is a bad trade.

So I put a gate on the client — but not a fixed one.

The gate

It lives in a pure decision function, called once per frame from the audio callback.

playing = (within 0.40 s of the last frame we heard ourselves emit)
 
if not playing:      pass            # never block a quiet speaker
elif peak < threshold: fill with silence  # residual echo
else:                pass            # a real interruption
                     3 in a row -> flush the local playback queue

Filling frames with zeros instead of dropping them matters. The server's speech detector expects a continuous stream and resets cleanly on silence. Removing frames outright skews its timing.

The threshold is not a constant — it's the larger of three times the learned echo floor and an absolute floor of 0.10.

floor     = EMA(peak of silenced frames, alpha 0.1)   # time constant ~0.4 s
threshold = clamp(floor * 3, 0.10, 0.30)

One rule matters most here. Frames that passed, and any period when we aren't playing, must not feed the floor. If it learns the user's voice or room noise as echo, the threshold runs away and eventually nobody can interrupt at all. The 0.30 ceiling exists for the same reason — even if the floor estimate spikes, a normal voice must always be able to break in.

Two traps

One: stopping a playback node can still fire completion callbacks for buffers that never played.

Right after an interruption flushes the queue, that callback stamps "we just made a sound," and the gate closes again for 0.40 s. That silences the user's first syllable. I disarm the clock on interruption and re-arm it when the next response is scheduled.

Two: stamp the playback time before hopping to the main thread. Stamp it after and the gate opens late by exactly the dispatch latency, which is enough for residual echo to slip through.

Verification

The gate is a pure function, so it has 21 unit tests. All three apps' full suites pass.

And the session-end diagnostics now carry how many frames the gate silenced, how many interruptions fired, and the learned floor. That's there to get evidence for the next tuning pass from real usage. It may be the most important part of the change — the thresholds are estimates, so I have to ship the data that grades those estimates.

Three-line self-check

Is there anywhere you assume hardware processing is "perfect"? Echo cancellation, noise suppression and auto gain are all probabilistic. There is a moment when the residual crosses your threshold.
Are you using a fixed threshold? The floor differs per device, per volume, per route. A constant is too aggressive on one device and inert on another.
Is your adaptive estimator learning something other than what it's meant to block? Feed passed signal into the floor and the threshold runs away.

The honest part

I could not verify this on a physical device. The bug doesn't reproduce in the simulator — you need a real speaker and a real microphone in the same room to produce residual echo. So every threshold in there is an estimate. 0.10, 0.30 and the 3× multiplier are values I chose, not values I measured.

Making the app learn its own threshold was the response to that, and whether that learning actually converges can also only be confirmed on device. Right now the telemetry is attached and I'm waiting for real usage data.

One thing is still undone. There's a call that tells the server where the user actually stopped hearing when they interrupt, but that code has diverged across the three apps so I deferred it. Today I only cut local playback while the server's conversation record stays intact — meaning the server believes the user heard everything.

If you work with realtime voice, log the mic peak while playback is running once. If it isn't zero, that's your app's own voice.