The power management of NVIDIA cards by linux driver(nouveau) is, tbh, pretty bad. I don't blame them. It's NVIDIA who is fucking us, and it's a tragedy that my anus is already raped too much.
Anyways, I need some stub VMs that are going to keep the GPU cool for a while. One can still install NVIDIA drivers on host-side, but I would not gamble with constant driver binding/unbinding, since it's unreliable and many things can get wrong. VFIO-pci module itself is rock solid at the current stage, so it's much nicer to let VFIO take control of those cards 24/7.
VGA passthru is also well covered by Proxmox wiki. God damn it. What's the point of leaving logs, if everything is this easy.
However, while passing through bootvga, I found a bug in Proxmox. It was doing everything except unbinding EFI framebuffer. I've already reported this.