One GPU, two models: sharing a card without the crashes

Once you start running AI on your own machine, a funny problem shows up: you end up with more than one thing that wants the graphics card, and they will not all fit at once. This is how I got dictation, a chatbot and an image generator to share a single card without them crashing into each other, and why the fix is scheduling rather than buying more hardware. The unfamiliar words are all explained on the back-to-basics page.

The problem, in plain terms

My server's graphics card has 32 GB of memory. That sounds like a lot until three programs want it:

• the dictation server, small, about 2 GB, on all day;
• a language model for writing and answering questions, large, over 20 GB when it is loaded;
• an image generator, which wants about 20 GB to itself.

Add those up and they do not fit. When they clash, the biggest one gets a memory error at the exact moment you asked it to do something. If you run one card and more than one AI program, you have this problem too, whatever the sizes. Here is what actually worked, learned the annoying way.

Rule one: always look before you debug

Before assuming anything is broken, look at what is on the card:

nvidia-smi --query-gpu=memory.used,memory.free --format=csv,noheader

Half of all "why did this run out of memory" mysteries end right here. Something you forgot about is holding 20 GB. In my case it was usually the image generator, which grabs the memory when it starts and does not give it back until it stops.

Rule two: decide who lives on the card and who visits

Not every program deserves to stay loaded. My split:

• Dictation stays. It is small, and dictation has to feel instant. A dictation server that takes twenty seconds to wake up is pointless.
• The chatbot visits. It is huge, but a few seconds to load at the start of a session is fine. Most tools already unload a big model after it sits idle for a while, which does most of the work for you.
• The image generator is an event. I start it for a batch of pictures and stop it afterwards. It never runs unattended, because it will not share while it is running.

Write your own version of that list. For each program the question is just: how annoying is it to wait for this to wake up, and how much memory does it hog?

Rule three: automate the eviction, and protect the important one

The bit that made it reliable was a small helper that sits in front of the chatbot. Before it loads the big language model, it checks how much card memory is free. If there is not enough, it tells the machine to unload whatever is sitting idle, waits, and then carries on.

The most important line in that helper is a protection rule: never unload dictation. Without it, the helper would happily evict the dictation server to make room for a chatbot question, and you would find out mid-sentence.

You do not need my exact helper. The idea is three simple steps, and each one is a single check:

1. before loading a big model, see how much memory is free;
2. if there is not enough, unload whatever has gone idle;
3. keep a "do not touch" list, and put dictation on it.

Rule four: measure before you spend money

My gut said the card was too small and I needed a second one. The measurements said the card was fine and the real problem was timing, not size. After the rules above, the same 32 GB now serves dictation all day, a 20 GB chatbot on demand, and image generation in scheduled bursts, with no crashes.

There is one real limit I found and kept: with everything loaded at once, there is not quite room for the very largest version of the chatbot. That is a genuine trade, and now it is a choice I made on purpose, rather than a crash I discovered by surprise. That is the whole difference this work bought me: the same hardware, but predictable.