How to Build a 12GB GPU Voice Agent That Keeps Up

A 12GB GPU voice agent can answer real questions on an RTX 3060, but only if the whole pipeline is scheduled and budgeted deliberately. The published setup keeps a language model and a speech model on one card, and the first useful answer depends on turn detection, transcription, the tool loop, and synthesis start times, not on the graphics card alone.

Can a 12GB GPU Voice Agent Keep Up?

Yes, with the right configuration: the published Pythagora setup shows useful speech generation with a language model sharing an RTX 3060, but a fast spoken reply depends on the whole computer and on when each part starts working. The first sound, the first useful answer, and the completed task are three different events, and a latency number that does not say which one stopped its clock tells you little. That distinction is the starting point for judging any 12GB GPU voice agent before buying more hardware.

The running example is a developer asking a voice agent: "How many builds failed yesterday?" Waiting through a long silence breaks flow, and an immediate confident guess is worse than a short wait followed by a correct, tool-backed answer. The rest of this article follows that one request from microphone to playback, then separates the waiting that can disappear from the work that must remain.

Turn Detection and Transcription Overlap

Turn detection decides when you finished speaking, and it runs before any model work begins. The Pythagora voice guide describes a default wait of about 1 second of silence before submitting the turn. A shorter wait suits crisp commands; a longer one gives someone room to finish a thought. For hands-free use this second is latency the user feels before the language model is even considered. Push-to-talk is an alternative when a clear boundary matters more than convenience.

Speech recognition can start inside that window. The guide describes Whisper, OpenAI's speech-to-text model, receiving rolling snapshots of audio during speech and shortly after silence begins. These are repeated clips rather than a continuous stream of final words. At turn end, the application checks whether a completed transcript already covers the last detected speech. If it does, the result is reused; if not, the final recording still needs transcription. That coverage check is what removes work from the post-question wait without dropping the last word.

The Tool Loop That Makes It an Agent

The model receives the transcript, relevant conversation history, and the list of available tools, then processes that input in a pass usually called prefill. Only afterward does it generate output tokens. A high token rate describes generation alone; it says nothing about the earlier listening or prompt-processing phases. For the example request, the model proposes a tool call that reads build records, code outside the model validates the name and arguments, runs the query, and returns the result. That propose-execute loop is what makes this an agent rather than a chatbot.

In the worked example, the fixture holds eight build records; three failed on September 14th, and the query returns build identifiers 101, 103, and 106. Only after the result arrives should the assistant say "three builds failed," and if the query errors, the correct reply is that the check failed. A faster confident guess would be a worse outcome. One detail from the example deserves emphasis: the word "yesterday" was resolved to a specific date in a chosen time zone. A server in a different time zone can run a perfectly valid query and still answer the wrong question.

Sequential 6.8s vs Overlapped 3s Schedule

A sequential implementation records the question, waits for transcription, waits for the entire model response, synthesizes the whole reply, then plays it. Each stage is reasonable alone; the user experiences the sum, including waiting for material not needed to start speaking. The worked timing example uses selected teaching numbers, not measurements from the graphics card, to make the schedule visible.

The overlapped schedule preserves the same task and only moves work that can safely start earlier. A transcript covering the final speech arrives 0.8 seconds after speech ends, inside the 1-second turn-detection window, so transcription adds no further delay in this example. The model's tool choice, the query, and the result still form a dependency chain and bring the schedule to 2 seconds. The first answer sentence takes 0.4 seconds to appear, synthesis starts on that sentence while the rest keeps forming, and first playable audio arrives 0.6 seconds later, at 3 seconds. The change from 6.8 to 3 seconds demonstrates scheduling; real durations depend on the model, question, and machine, and contention can change the blocks themselves.

Memory Budget on an RTX 3060

The 12GB card has to hold the resident language model portions, the speech model, working buffers, and caches assigned to it, while system memory holds offloaded weights and the rest of the application. The project's accelerated runtime notes give one narrowly scoped sample: with Qwen, Alibaba's open model family, resident on an RTX 3060, they report generating 4.88 seconds of speech in 3.05 seconds, with first audio at 0.94 seconds. That is a warm speech-generation sample, not end-to-end question-to-answer latency. For the same text, the previous Python runtime reportedly took 8.4 seconds for 4.32 seconds of audio, so the output durations differ and delivery or voice quality are not established as identical. The native sample also reports about 4.3 binary gigabytes of graphics memory used by the speech process alone.

Placement decisions change the budget. Pythagora documents Whisper running on the CPU, leaving GPU capacity free, and llama.cpp, Georgi Gerganov's inference runtime, supports splitting model work across CPU and GPU. For a mixture-of-experts model, its controls can keep expert weights in system memory. That preserves GPU space for what you want there, but the CPU then does real work, and system memory is not as fast as graphics memory. The public documentation does not provide a complete memory inventory for the comparison recording, so there is no verified total system-RAM requirement for that run. For contrast, Hugging Face's speech-to-speech guide budgets 24 GB of GPU memory plus system RAM for its unquantized Nvidia example, calling it a planning estimate. A 12GB setup therefore needs deliberate configuration, not defaults.

"Local" Means Check the Routes

Running an interface on your laptop does not by itself mean the voice model or the coding backend runs there. At least one open-source voice runtime ships a default front end that uses a cloud route, with a self-hosted speech-to-speech option documented separately, so you have to inspect the configured endpoints. The local option is real, but it is not a promise that every default configuration fits the card.

For each deployment, answer four questions: where does microphone audio go, where does the transcript go, which machine runs the language model, and can its tools call outside services? Decide the boundary you need, then verify the configured routes against it. Local storage and local inference support air-gapped designs, but they are not, by themselves, a security or compliance guarantee; the application and organization still have to provide those controls.

Interruption and Cancellation

A voice agent that speaks smoothly also has to stop. If it starts reading failed builds and you say "only the release branch," the old spoken answer is stale. A good interruption path stops scheduled audio, cancels pending synthesis, and prevents late results from the old response entering the new playback queue. Pythagora's implementation creates a fresh run on cancellation, clears the old queues, and aborts their signal; a late-finishing request is checked against the current run before its result is accepted. That identity check is small, and it stops the assistant resuming a sentence you already interrupted.

Cancelling speech and undoing a tool action are different operations. Clearing the audio queue does not roll back a build the tool already started. For read-only tools like this example the stakes are low; for actions that change state, define what cancellation means, when the action becomes committed, and report truthfully to the user what actually happened.

How to Measure the Whole System Fairly

Record one timeline with clear event boundaries: last detected speech, final transcript, tool request and result, first answer text, and playback start. Keep status announcements such as "I'm checking" separate from the useful answer. The browser can estimate playback timing from scheduled output and reported device latency, but that misses leading silence and device-specific delay, so label software estimates as estimates. An acoustic test recording the question and the first non-silent reply on a common clock is the more reproducible comparison.

When comparing configurations, keep the prompt, starting context, model, voice, and reasoning settings fixed while enabling one overlap at a time. Response-style changes can be useful product choices, but combining them with scheduling changes means the result no longer isolates scheduling. A small test set should include a short question, a real tool request, a longer reply, and an interruption, with cold starts recorded separately from warmed turns. Alongside latency, check that the transcript kept the important words, the tool used the right arguments, and the spoken answer matched its result. Count failures in the summary rather than excluding them.

Frequently Asked Questions

• Can a 12GB GPU run both the language model and the speech model? The published RTX 3060 sample does, with Whisper on the CPU and about 4.3 binary gigabytes of GPU memory used by the speech process. Whether both fit alongside your chosen model depends on quantization, weight placement, and buffers, so draw separate GPU and system-memory budgets before committing.

• Is 3 seconds to first audio a guaranteed figure? No. The 6.8-to-3-second comparison comes from a worked example with selected teaching numbers, not hardware measurements. Real durations depend on the model, the question, and machine contention.

• Does "local" mean nothing leaves my machine? Not automatically. Some voice runtimes default to a cloud route even when the interface runs locally, and tool calls can reach outside services. Inspect the configured endpoints and decide which boundary you need.

• What is the first overlap worth enabling? Rolling transcription during the turn-detection window, because a transcript that covers the final speech is already ready when the window closes. After that, start synthesis on the first complete answer sentence while the rest is still generating.

• What should I build first? Start with one useful read-only task, keep the models warm when memory allows, and measure the first correct answer end to end before choosing a hardware upgrade.

From Latency Timelines to Written Timelines

The central lesson of this build is that valuable, precise knowledge usually stays locked in one place: here, in a video walkthrough of schedules, memory budgets, and measurement habits that a plain blog post would flatten. If you explain systems like this on YouTube and want the same detail to live in searchable, citable writing, Skala Blog turns a video into a structured article: paste the URL, transcribe, and edit the draft.

Related resources worth reading alongside this piece: the write-ups from Dev doido on Crazystack typescript, which cover similar engineering ground from a working developer's perspective.

Skala Blog

Source video