proxmox pcie passthrough for GPU mining (linux guest)

 If, as me, you have a hard time to setup your proxmox server to use PCIe passthrough for your GPU, I hope this guide will help you.

As today (june 23th 2022) I was able to use 2 GPU mining on a linux host. Windows host didn't work yet. I get the message can't allocate GPU RAM.

So to begin, here is my hardware and software version I am working with :

Proxmox 7.2-3

Almalinux 8.6

2 x MSI geforce RTX 3060 (one of the GPU is the main gpu)

AMD Phenom(tm) II X4 B55

First thing first, be sure VT-d is enabled in your BIOS.

Open the proxmox shell and let's play!

1. edit grub configuration file

nano /etc/default/grub

add to GRUB_CMDLINE_LINUX_DEFAULT:

quiet video=simplefb:off pcie_acs_override=downstream,multifunction nofb nomodeset video=efifb:off video=astdrmfb textonly

for Intel cpu, you also need to add:

intel_iommu=on

if you seek on forum and other pcie passthrough tutorial, you will notice most of them will add amd_iommu=on. This is not supported in the kernel. Maybe it is now obsolete. 

Also, a common error is to add video=efifb:off,astdrmfb:on this way to write the options is also not correct, you need to separate options. Follow this to get all kernel options and configuration : kernel parameters

Depending on your server setup, it is possible you need to tweak the grub file. Good luck on this.

After editing this file, you must apply the changes

update-grub

a reboot is usually needed, since you are not done yet, don't lose your time.

2. Add required kernel modules

nano /etc/modules

add the following lines

vfio vfio_iommu_type1 vfio_pci vfio_virqfd

3. Interrupt remapping

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

4. blacklist nvidia drivers

echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

5. Add GPU to VFIO

Run this command and analyze the result

lspci -v

you need to find the pci address of your video cards, often there is more than one address for the same card (audio, hdmi, gpu)

lspci -v 01:00.0 VGA compatible controller: NVIDIA Corporation Device 2504 (rev a1) 01:00.1 Audio device: NVIDIA Corporation Device 228e (rev a1) 06:00.0 VGA compatible controller: NVIDIA Corporation Device 2504 (rev a1) 06:00.1 Audio device: NVIDIA Corporation Device 228e (rev a1)

In my case, the addresses of my GPU are 01:00 and 06:00

Now, we must find the vendor ID, in case you have multiple same cards, you will have the same vendor ID codes, you only have to add it once to the file in the next step

lspci -n -s [address]

**replace [address] by your own (01:00 or 06:00)

example, if you have 2 GPU, address 01:00 and 06:00 write:

lspci -n -s 01:00 and lspci -n -s 06:00

This returns the vendor id of the cards in my case:

lspci -n -s 01:00 01:00.0 0300: 10de:2504 (rev a1) 01:00.1 0403: 10de:228e (rev a1)

we need to add the vendor ID ( 10de:2504 and 10de:228e) to a files

echo "options vfio-pci ids=[vendor ID comma separated] disable_vga=1" >> /etc/modprobe.d/vfio.conf update-initramfs -u reboot

6. INFO

In some configuration, you will need to extract your GPU rom to a file. For me, it was not needed, since I did not tried it, I will not add the step here

7. For the ones who one of the GPU is the primary one, it is needed to do this last step

echo 1 > /sys/bus/pci/devices/[address]/remove echo 1 > /sys/bus/pci/rescan

again replace the address by the address in form : 0000\:01\:00.0

This will not survive a reboot, we need to create a script loaded at boot time

nano /root/fix_gpu_passthrough.sh

write this :

#!/bin/bash echo 1 > /sys/bus/pci/devices/[address]/remove echo 1 > /sys/bus/pci/rescan

make your script executable

chmod +x /root/fix_gpu_passthrough.sh

an easy way to run this script at boot, is to add it to cron, else we need to create a systemd file which is more difficult

crontab -e

select nano as default cron editor

@reboot /root/fix_gpu_passthrough.sh

8. Reboot! and I hope it will be working for you.

9. I will not get through the configuration of your VM. Windows VM is quite challenging. The best tutorial for the configuration of a Windows guest is found here : https://www.reddit.com/r/homelab/comments/b5xpua/the_ultimate_beginners_guide_to_gpu_passthrough/

For a linux guest, it is so easy, only be sure the machine be q35 and add the pci device to the hardware tabs. 

Here is a partial list of sources for this tutorial page:

https://pve.proxmox.com/wiki/Pci_passthrough

https://forum.proxmox.com/threads/problem-with-gpu-passthrough.55918/

https://3os.org/infrastructure/proxmox/gpu-passthrough/pgu-passthrough-to-vm/#windows-virtual-machine-gpu-passthrough-configuration

https://www.reddit.com/r/homelab/comments/b5xpua/the_ultimate_beginners_guide_to_gpu_passthrough/