GPU passthrough has a reputation for being a temperamental piece of configuration. It is deserved, but for a specific reason: six conditions have to be met at the same time, and when one is missing, the symptom almost never points at the right cause.
Taken in order, these obstacles clear methodically.
First of all: do you need it?
Settle this one first, because it sometimes saves you all the rest.
Passthrough dedicates the card to a single virtual machine. The host loses it, and so do the other guests. It is an exclusive model.
If what you need is video transcoding or inference shared across several services, a container with device access answers better: several containers can use the card at the same time, the host keeps access to it, and the configuration is vastly simpler.
Passthrough earns its place for exclusive use: gaming, a virtualized workstation, a different operating system, an application that demands the full vendor driver.
Trap 1: the IOMMU is off
The IOMMU is the hardware component that makes it possible to isolate a device's memory accesses. Without it, no passthrough is possible.
Two separate activations are required, and you need both:
- In the motherboard firmware. The setting goes by different names depending on the vendor and often hides in the advanced chipset options. It is frequently disabled by default.
- At kernel boot, through a parameter passed to the bootloader.
Verification is simple: at boot the kernel logs IOMMU detection and the composition of the groups. If those messages are absent, stop right there, because nothing that follows will work.
Trap 2: the indivisible group
This is the most structural blocker, and the one that no amount of software configuration can do anything about.
Devices are split into IOMMU groups according to what the hardware is capable of isolating. Passthrough operates at the granularity of the whole group, never of a single device.
If your graphics card sits alone in its group, all is well. If it shares that group with a network controller, a storage port or a USB controller, all of it has to be handed to the virtual machine, including components the host needs in order to run.
How good that separation is depends on the motherboard and on the slot you use. Entry-level platforms group things broadly; server and high-end platforms isolate finely.
Two possible answers: move the card to a different slot, which is sometimes enough to land it in a friendlier group, or fall back on a kernel workaround that forces a finer split, bearing in mind that it weakens the very isolation it claims to provide, and that it only belongs on a machine where you control every guest.
Trap 3: the host driver will not let go
At boot, the system detects the card and loads its graphics driver. Once it has hold of it, the card is no longer available for passthrough.
So you have to prevent that, before the driver even loads: either by putting the driver in question on a blacklist, or by explicitly binding the device ID to the assignment driver from system initialization onwards.
Diagnosis is direct: in the device list, look at which driver is attached to the card. If it is the graphics driver, the host has taken it. If it is the assignment driver, it is ready.
Trap 4: the card that drives the console
A particularly nasty case. A card initialized by the firmware at boot, and used for the host's display, resets badly once detached.
The symptom is characteristic: the virtual machine boots once and works. You shut it down, you start it again, and nothing, until you physically reboot the machine.
The most reliable fix is to give the host another video output: integrated graphics on the CPU, or a second modest card. The card you want to pass through is then never touched by the firmware nor by the host, and behaves correctly.
Trap 5: the ROM you need
When the card you want to pass through is also the one that boots the machine, it has already been initialized by the firmware. The virtual machine, trying to initialize it in turn, fails.
The workaround is to hand the VM a copy of the card's ROM, unaltered by boot-time initialization. You extract it either from the system, or from a public database matching the exact model.
This is only necessary in that specific configuration. With a separate video output for the host, the problem does not arise, which is one more reason to prefer that solution.
Trap 6: the reset that never happens
The last obstacle, purely a hardware one: some cards do not reset correctly when the virtual machine shuts down. They stay in an inconsistent state, and the next boot fails.
It depends on the model and the generation. Workarounds exist, none of them universal.
The practical dodge: configure the virtual machine to start with the host and never stop it in normal use. It is not elegant, but on a virtualized workstation that runs permanently, the constraint is acceptable.
Diagnostic order
- Is the IOMMU detected at kernel boot? If not, nothing else is relevant.
- Is the card alone in its group? If not, change slots before any other effort.
- Which driver is attached to the card? It should be the assignment driver.
- Does the host have another video output? If not, expect traps 4 and 5.
- Does the VM boot once and then never again? That is the reset problem.
Worked through in that order, these points cover very nearly every failure. And if you are blocked at point 2 on a motherboard that groups everything together, the honest conclusion is sometimes that a container with device access will serve your real need better.
Comments