How Real-Time Speech Recognition Works (For Non-Technical Streamers)

You speak, and 1.4 seconds later, the words appear in Spanish at the bottom of your stream. What actually happens in that 1.4 seconds? This guide explains the full pipeline — from microphone input to translated subtitle — in plain English, with no computer science degree required.

The Big Picture

Real-time speech-to-subtitle translation is a 5-step pipeline. Each step adds a small amount of time; the total is what you experience as "subtitle delay." Modern systems like StreamTranslate optimize each step to keep the total under 2 seconds.

Step 1: Audio Capture

The browser source in OBS creates a web page that runs inside your streaming software. This page has access to your microphone input (via Web Audio API) and streams the raw audio data over a WebSocket connection to StreamTranslate's servers. The audio is transmitted as compressed chunks every 250–500 milliseconds — small enough to feel real-time, large enough for the AI to process meaningfully.

This step adds: ~50–100ms latency (network transmission time).

Step 2: Automatic Speech Recognition (ASR) with Deepgram

Deepgram is an AI company specializing in fast, accurate speech-to-text. Their Nova-2 model is specifically optimized for low-latency transcription — meaning it's designed to produce text output as quickly as possible, rather than waiting for a full sentence before outputting.

Deepgram processes the audio stream using a deep neural network. The model has been trained on hundreds of thousands of hours of speech — including gaming content, streams, podcasts, and everyday conversation. It produces a probability-weighted prediction of what was said, outputting words as it becomes confident in them.

This is why you sometimes see the subtitle text "flicker" — the model outputs an initial guess, then updates it as more context arrives. This is called "word finalization" and is a normal part of real-time ASR.

This step adds: ~200–400ms latency.

Step 3: Neural Machine Translation

Once a phrase or sentence is transcribed, it's passed to a translation model. Modern NMT (neural machine translation) systems have replaced the older "phrase substitution" approach with transformer-based models that understand context. "I'm going to smoke them" means something very different in a gaming context than a general context — modern NMT handles this better than earlier systems, though not perfectly.

Translation happens in ~100–200ms for common language pairs (English to Spanish, French, Portuguese). Less common pairs (English to Japanese, Korean) may take slightly longer due to structural differences in the target language.

This step adds: ~100–300ms latency.

Step 4 and 5: Display and Composite

The translated text is pushed via WebSocket to your browser source overlay. JavaScript renders it as styled HTML text on a semi-transparent background. OBS composites this layer onto your video output during encoding. This happens in real time and adds ~50–100ms total.

Total Latency Breakdown

Why It Runs in the Cloud

Running speech recognition and translation on your local PC would require GPU-accelerated hardware most streamers don't have, and would compete with your game for resources. Cloud processing offloads the compute entirely — your PC only handles the lightweight audio capture and text rendering. This is why StreamTranslate has essentially zero CPU impact on your streaming machine.

For more on latency in live streaming specifically, see: latency in live stream captions — what's acceptable?