Home lab serverPart 8 of 13

Build note

Adding a third GPU moved the target

The transcription model was supposed to run on the RTX 3060. After a third card was installed, it loaded on the 2 GiB Quadro P400 instead. The setting had not changed. The hardware order had.

Before the P400 was added, the 3060 sat at PCI address af:00.0, and the Vulkan visibility setting used index 1 for the target. After the P400 went into that slot, the 3060 moved to d8:00.0; af:00.0 now belonged to the P400. The old number still selected a GPU. It just selected the wrong one.

Three tools, three orders

With three NVIDIA cards present, the machine reported:

View Order
nvidia-smi 0 = RTX 3090, 1 = Quadro P400, 2 = RTX 3060
Vulkan 0 = RTX 3060, 1 = RTX 3090, 2 = Quadro P400
Hex device list 0 = RTX 3090, 1 = Quadro P400, 2 = RTX 3060

The numbers are not interchangeable. GGML_VK_VISIBLE_DEVICES follows the Vulkan enumeration in this case. CUDA_VISIBLE_DEVICES belongs to CUDA. Hex also has its own device list. The label GPU 1 is incomplete unless I say which tool printed it.

The log showed what actually happened

Hex’s process log named the physical device as well as the backend. Before the P400 was added, transcription loaded on the RTX 3060 at 0000:af:00.0. After the card moved, the same address named the Quadro P400. The stale index sent the model to the 2 GiB card, where it could not fit reliably.

The fix was to re-enumerate with hex model devices, then set GGML_VK_VISIBLE_DEVICES=2 for the 3060 in Hex’s device list. For the other Hex renderer, the configuration used the full adapter name instead of a number. The lesson was not “use index 2 forever.” It was to identify the device by UUID, PCI address, or a stable adapter name, and verify the application’s own log after loading.

The rule that would have prevented the next week

After adding or moving a card, re-check every numeric GPU setting. Read the device order from the tool that consumes the setting, then confirm the loaded model’s physical device from the application log. An index that still parses can still be wrong.

How this was made

  • Written by: Rupesh, from GPU enumeration output, the Hex service drop-in and its model-load log, with AI drafting and editing.
  • Device addresses and ordering: from nvidia-smi, Vulkan and hex model devices captures after the P400 was installed.
  • AI-assisted sections: structure and copy editing.