Task 014: Proxmox VGA Passthru

Proxmox allows VMs to directly access PCI devices (so-call "pass-through"). It provides UI for passthrough, but its wiki also provides necessary details.

So, in this log, I'm just writing down what "I" had to do.

My Setup

  • I'm passing through boot VGA, the VGA used as the main display during boot process. This is a special case, because VGA firmware is already initialized and modified in-memory during boot. Because of this, you need to feed QEMU the original firmware.

  • My VGAs are from NVIDIA, which notoriously bans VMs through their Windows drivers. Every time I think about, I can't stop hating and cursing NVIDIA. But, these days, AMD's performing well, while NVIDIA is shitting their pants. Really, I should have bought AMD, but they ran too hot for my small form factor cases.

Obtaining NVIDIA firmware

There's a website1 where people upload their VGA firmwares, mainly for performance tuning, and some unlocking features thru black hat magic.

Note that, technically, running firmwares uploaded by random guys might have some security implications. But, practically, there's absolutely zero merit in planting random code in your VGAs... at least yet.

Extracting actual VBIOS

The thing about NVIDIA firmware is that it's, in fact, a firmware "package" in NVIDIA specific format. IIRC, the package is consumed by firmware updaters, but not sure if you still get this by dumping firmware directly from the card.

Anyways, there's also a tool for extracting the actual VBIOS from the package:

Feeding VBIOS to QEMU

Once VBIOS is ready, this should be passed to QEMU. QEMU VFIO PCI device does have option romfile for passing firmwares.

-device vfio-pci,<...>,romfile=/path/to/some.rom

On Proxmox, this has to be done by manually editing the config file:

hostpci0: 01:00,romfile=/path/to/some.rom

Note that ROMBAR should be enabled (which is by default). The card will still work, but might have some issues.

Setting up Proxmox VM

If it were an up-to-date QEMU with libvirt, you only need to:

  1. Use OVMF firmware.
  2. KVM signiture hiding (kvm=off, vendor=)

... but, Proxmox uses an older QEMU w/ many patches in both kernel and QEMU. I'm not sure what's the exact cause, but NVIDIA driver raises "Error 43" if VM is running on i440fx machine. If you're running Proxmox, you MUST use q35 machine type, with PCI-E enabled.