Moving Whisper to the GPU, and the CUDA trap

The dictation server I built works, but on a plain computer it is slow. A five-second clip took about twelve seconds to come back. Fine for a voice note, useless for anything that should feel instant.

Then I gave it a graphics card, and the same clip came back in under half a second. That is the difference between "a project" and "the thing I use every day". This is how I did it, why it is so much faster, and the one trap that wasted an evening so it does not waste yours. New words are explained on the back-to-basics page.

Why a graphics card makes such a difference

A graphics card, or GPU, was built for games, but it turns out the maths behind games and the maths behind AI are the same shape. So the card that draws explosions quickly also transcribes speech quickly. The processor in your computer does this work one lump at a time; the graphics card does thousands of small pieces at once.

The catch is memory. A card has its own memory, called VRAM, and the speech model has to fit inside it to run fast. The dictation model is small, so this is easy: it uses about 2 GB, and most cards have more than that spare.

What you need

• The dictation server from the first guide, already working.
• An NVIDIA graphics card and its drivers installed.
• The NVIDIA container toolkit, which is one package that lets Docker reach the card. Install it, then carry on.

The easy version

The change from the first guide is small. You swap two settings and point at the graphics-card version of the program:

sudo docker run -d --name whisper \
  --gpus all \
  -p 9000:9000 \
  -v /srv/whisper/data:/var/lib/whisper \
  -e WHISPER_MODEL=large-v3-turbo \
  -e WHISPER_DEVICE=cuda \
  -e WHISPER_COMPUTE=float16 \
  -e WHISPER_API_KEY=$(cat ~/whisper-api-key.txt) \
  --restart unless-stopped \
  hwdsl2/whisper-server:cuda

In plain terms: use the card (--gpus all), and because the card is so much faster, upgrade to the biggest, most accurate model rather than the small one. On most machines that is all you need, and dictation is now effectively instant.

The trap that cost me an evening

On one machine this worked first time. On another, with the same drivers, the program refused to start and kept restarting with this message:

CUDA forward compatibility was attempted on non supported HW

The obvious reading is "your drivers are too old, update them". That is wrong, and chasing it wasted my evening. The real cause is a mismatch between the version the program was built for and the version your card's drivers speak. There is a mechanism meant to bridge that gap automatically, but it only works on the expensive data-centre cards, not on the ordinary gaming cards most of us own. Same drivers, same program: the data-centre card runs it, the gaming card refuses.

So if you see that message on a normal graphics card, stop updating drivers. No driver update will fix it. You need a version of the program built for the version your card already speaks.

The fix

Check the version your card supports (it is in the top corner of the output of nvidia-smi). Then use, or build, a version of the Whisper program made for that. In my case I built a small custom version on the matching base, tagged it, and pointed the run command at it instead. The card that had refused the standard version ran the custom one happily, on the graphics card, at full speed.

If you are not comfortable building a custom version, this is the point to ask in the forum; it is a common enough snag that someone will have the matching build for your card.

The numbers, measured not guessed

Same five to eleven second clip, same machine:

Setup	How long the answer took
Processor only	about twelve to fourteen seconds
Graphics card	under half a second

That is not a small improvement, it is a different experience. Below half a second, the server stops being the slow part. You are.

What I would tell you

1. Get the processor version working first. It proves everything end to end with nothing to go wrong.
2. If you have any NVIDIA card, try the graphics-card version next. If it starts, you are done, and dictation is now instant.
3. If it keeps restarting with the compatibility message and you have an ordinary gaming card, do not fight the drivers. Use a version built for your card and move on with your evening intact.