Build note
The 12 GiB card, the 24 GiB card, and the index
One LM Studio run used 20.7 GiB across two cards. That number is easy to
misread as the weights of a model that was too big for the 24 GiB card. It
wasn’t. The run used qwen3.5-9b with a 262,144-token context and four parallel
slots. The model was split across the cards, and the 3060 was not driving the
display during the measurement.
The split model
LM Studio reported the model itself at 6.55 GiB. GPU memory use was 12.2 GiB on the 3090 and 8.6 GiB on the 3060, 20.7 GiB combined. The 3060 was idle as a display card for that run.
The context window and parallel slots were configured much larger than a small local test needs. They affect how much memory the runtime reserves. The GPU figures show where that run allocated memory; they do not prove which setting caused each allocation. The safe conclusion is narrower: this particular configuration used both cards, and the display was not active on the 3060.
What model size does and does not tell you
Parameter count is only part of the memory requirement. Precision, quantization, context length and runtime overhead matter too. A rough starting point for the weights is:
| Model configuration | Approximate weights | What the cards suggest |
|---|---|---|
| 7B at 4-bit | 3.5–5 GiB | Often fits on 12 GiB, depending on context and overhead |
| 7B at 16-bit | About 14 GiB | Fits on the 24 GiB card; too large for 12 GiB |
| 13B at 16-bit | About 26 GiB | Exceeds 24 GiB before context memory; quantize or split |
| 20B at 4-bit | About 10 GiB | May fit on 24 GiB; actual quantization and context decide |
These are estimates, not promises. The model file is not the whole runtime footprint. A long context or several parallel requests can use more memory than the weights alone suggest.
Three tools, three indices
The two cards do not have one shared device number. Each tool enumerates them in its own order:
| Tool | Order in the two-card setup |
|---|---|
nvidia-smi |
0 = 3090, 1 = 3060 |
| Vulkan | 0 = 3060, 1 = 3090 |
| Hex “auto” | Picks the 3090, because it has the most VRAM |
The app’s “auto” is a poor choice for a transcription job when the 3090 is reserved for an LLM. It prefers the card with the most VRAM, so the small job can land on the large-model card. The same setting name can also be interpreted against an application’s own device list rather than the Vulkan list.
The correction is not to copy 0 or 1 from another tool. I check the app’s
device list and the model-load log to confirm the physical card. A PCI address
or UUID is more explicit when the application supports it. Part 8 shows what
happened after a third GPU changed those lists.
The rule that would have prevented the next week
Treat context and parallel slots as part of the model’s memory budget. Then verify the selected physical card in the application’s own log. A model name and an index do not tell me where it actually loaded.
How this was made
- Written by: Rupesh, from his
nvidia-smioutput, GPU role notes and LM Studio run, with AI drafting and editing. - Numbers:
nvidia-smi(24576 MiB and 12288 MiB, driver 595.99.02, CUDA 12.8.93) and LM Studio’sqwen3.5-9brun at 262,144 context and four parallel slots. - AI-assisted sections: structure and copy editing.