Proxmox_GPU_Passthrough/README.md

SOP: AMD Radeon AI Pro R9700 GPU Passthrough on Proxmox VE

Repository: pch/Proxmox_GPU_Passthrough
Target: A Proxmox VE compute node dedicated to AI/ML GPU passthrough
GPU: AMD Radeon AI Pro R9700 32GB (1002:7551)
GPU Audio Function: AMD Navi 48 HDMI/DP Audio Controller (1002:ab40)
Date: 2026-08-26
Goal: Isolate the R9700 from the Proxmox VE host and bind it to vfio-pci so it can be passed through to a VM for AI/ML workloads.

---

1. Discover the GPU on the node

Log in to the Proxmox node via SSH and identify the GPU and its audio function.

ssh root@<proxmox node>
lspci -nn | grep -i 'AMD'

Expected output (excerpt — values will vary by system):

<pp>:00.0 PCI bridge [0604]: AMD/ATI Navi 10 XL Upstream Port of PCI Express Switch [1002:1478]
<pp>:00.0 PCI bridge [0604]: AMD/ATI Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
<pp>:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
<pp>:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]

Where <pp> is the PCI domain/bus/slot address assigned by the host (e.g. 84:00).

Key facts:

• GPU core vendor/device ID: 1002:7551.
• GPU audio vendor/device ID: 1002:ab40.
• Link speed should be Gen5 x16 (Speed 32GT/s, Width x16).

---

2. Verify prerequisites

2.1 IOMMU is enabled

dmesg | grep -iE 'AMD-Vi|IOMMU'

Expected output:

AMD-Vi: Extended features (...): PPR X2APIC NX GT IA GA PC GA_vAPIC
AMD-Vi: Interrupt remapping enabled
AMD-Vi: X2APIC enabled
AMD-Vi: Virtual APIC enabled

2.2 CPU virtualization extensions are present

grep -iE 'svm|npt|avic' /proc/cpuinfo | head -2

Expected flags include svm, npt, avic.

2.3 IOMMU grouping is clean

for g in /sys/kernel/iommu_groups/*; do
  for d in $g/devices/*; do
    echo "$(basename $g): $(basename $d) $(lspci -nns $(basename $d))"
  done
done | grep -iE '1002:7551|1002:ab40'

Expected output (group numbers and addresses will vary):

<gpu group>: 0000:<pp>:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
<audio group>: 0000:<pp>:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]

Both functions should be in their own IOMMU groups. If they are the only devices in their respective groups, they can be passed through cleanly without needing the ACS override patch.

2.4 Check current driver binding

lspci -nnk | grep -iE '1002:7551|1002:ab40' -A 3

Before configuration, the output will look like:

<pp>:00.0 VGA compatible controller [0300]: ... Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
<pp>:00.1 Audio device [0403]: ... Navi 48 HDMI/DP Audio Controller [1002:ab40]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

The GPU must be freed from both drivers before vfio-pci can bind.

---

3. Configure the host for VFIO passthrough

All commands below run as root on the Proxmox node.

3.1 Enable IOMMU passthrough mode in GRUB

Edit /etc/default/grub:

sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"/' /etc/default/grub

Resulting line:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

• amd_iommu=on: Explicitly enables the AMD IOMMU (AMD-Vi) at boot.
• iommu=pt: Enables IOMMU passthrough mode, meaning DMA is only translated for devices assigned to VMs, reducing host overhead and avoiding conflicts between host drivers and the IOMMU.

3.2 Load VFIO modules at boot

Create /etc/modules-load.d/vfio.conf:

cat > /etc/modules-load.d/vfio.conf <<'EOF'
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
EOF

3.3 Blacklist the host GPU and audio drivers

Create /etc/modprobe.d/blacklist-amdgpu-r9700.conf:

cat > /etc/modprobe.d/blacklist-amdgpu-r9700.conf <<'EOF'
blacklist amdgpu
blacklist snd_hda_intel
EOF

Why blacklist both?

• amdgpu is the Linux kernel driver for AMD GPUs. If it loads, it will claim the R9700 and prevent vfio-pci from binding.
• snd_hda_intel is the generic High Definition Audio driver. It will claim the HDMI/DP audio function of the same AMD GPU (<pp>:00.1). For clean passthrough, both functions must be free.

3.4 Bind VFIO to the specific PCI IDs

Create /etc/modprobe.d/vfio.conf:

cat > /etc/modprobe.d/vfio.conf <<'EOF'
options vfio-pci ids=1002:7551,1002:ab40
EOF

This approach is targeted: only the R9700 GPU and its audio function are captured by vfio-pci. Other AMD GPUs, if added later, are not affected.

3.5 Regenerate bootloader and initramfs

update-grub
update-initramfs -u -k all

---

4. Reboot and verify

4.1 Reboot the node

reboot

4.2 Verify kernel command line

cat /proc/cmdline

Expected:

BOOT_IMAGE=/boot/vmlinuz-... root=... ro quiet amd_iommu=on iommu=pt

4.3 Verify VFIO modules loaded

lsmod | grep -iE 'vfio|amdgpu'

Expected: vfio_pci, vfio_pci_core, vfio_iommu_type1, vfio, and iommufd present. amdgpu should not be loaded.

4.4 Verify device binding

lspci -nnk | grep -iE '1002:7551|1002:ab40' -A 3

Expected:

<pp>:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu
<pp>:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

Note: The Kernel modules: line may still list amdgpu and snd_hda_intel. That is normal — it only means those drivers could drive the device. The important value is Kernel driver in use: vfio-pci.

4.5 Verify sysfs binding

ls /sys/bus/pci/drivers/vfio-pci/ | grep ':'

Expected:

0000:<pp>:00.0
0000:<pp>:00.1

---

5. Proxmox VM configuration (when ready)

When you create the VM that will receive the R9700, use these settings:

Setting	Value
Machine type	q35
BIOS	OVMF (UEFI)
CPU type	host
CPU topology / NUMA	Match host only if AI workload benefits; otherwise default
Memory	Static allocation recommended for AI workloads; disable ballooning
Display	Standard Proxmox console (VirtIO/VNC) — do not set R9700 as primary
PCI passthrough device 1	0000:<pp>:00.0 — Radeon AI PRO R9700
PCI passthrough device 2	0000:<pp>:00.1 — HDMI/DP Audio
PCI options	Primary GPU: No; PCI-Express: Yes; ROM-Bar: Yes (default is usually fine)

Inside the guest, install the appropriate AMD driver stack (e.g. ROCm on Linux, or Adrenalin/amdgpu driver on Windows) for AI workloads.

---

6. Lessons learned and important notes

6.1 Why both GRUB flags are needed

• amd_iommu=on: This is the on/off switch for AMD IOMMU. Without it, the kernel has no device isolation layer and cannot safely hand a PCIe device to a VM.
• iommu=pt: This enables passthrough mode. It tells the IOMMU to leave host DMA untranslated and only translate DMA for devices assigned to guests. This reduces host overhead and avoids IOMMU conflicts with the host.

Using only amd_iommu=on can sometimes work, but iommu=pt is the Proxmox-recommended setting for GPU passthrough hosts and avoids subtle bugs.

6.2 Why both amdgpu and snd_hda_intel must be blacklisted

A modern GPU is not one PCI device — it is multiple functions on the same card:

• <pp>:00.0: the GPU itself (video/compute).
• <pp>:00.1: the HDMI/DisplayPort audio controller.

The amdgpu kernel driver will bind to <pp>:00.0. The snd_hda_intel driver — despite the name, this is the generic HDA audio driver used by AMD, Intel, and NVIDIA GPUs — will bind to <pp>:00.1.

If either function remains claimed by the host, the passthrough will be incomplete or fail. Blacklisting both drivers ensures the entire card is free for vfio-pci.

6.3 Why the audio function matters even for AI-only workloads

Even if the VM will not output audio, passing through the audio function alongside the GPU is recommended because:

1. It prevents the host from retaining a claim on part of the card.
2. Some AMD GPU drivers expect both functions and behave more reliably when passed together.
3. The audio function is in its own clean IOMMU group, so it adds no risk.

6.4 Use PCI IDs, not just a driver blacklist

The blacklist alone prevents the host drivers from loading, but it does not tell vfio-pci which devices to claim. The options vfio-pci ids=1002:7551,1002:ab40 line explicitly binds the R9700 to VFIO. Using PCI IDs is more precise than a blanket blacklist and is safer if more GPUs are added later.

6.5 Why a reboot is mandatory

The amdgpu driver was already bound to the R9700 at the time of configuration. You cannot cleanly unbind a GPU driver that has initialized firmware, memory, and display engines without a reboot. A full reboot guarantees the new GRUB options, module load order, and blacklist take effect before any driver can claim the card.

6.6 BMC / host console video

Always verify the host has an alternative console (e.g. an ASpeed BMC VGA controller, a separate low-end GPU, or serial console) before blacklisting the primary GPU driver. Blacklisting amdgpu must not remove the host's ability to display video or log in locally.

6.7 IOMMU groups are clean, no ACS override needed

For this setup, the GPU was the only device in its IOMMU group and the audio function was the only device in its group. This is the ideal scenario for passthrough — no ACS override patch or risky kernel parameters are needed. If your system groups the GPU with other devices, additional steps may be required.

6.8 Do not create the VM as part of this SOP

This SOP intentionally stops at host preparation. VM creation, guest OS installation, and driver installation are separate steps and should be documented in their own procedure.

---

7. Files changed on the Proxmox node

File	Purpose
/etc/default/grub	Enables IOMMU and passthrough mode at boot
/etc/modules-load.d/vfio.conf	Loads VFIO framework modules at boot
/etc/modprobe.d/blacklist-amdgpu-r9700.conf	Prevents host drivers from claiming the R9700
/etc/modprobe.d/vfio.conf	Binds vfio-pci to the R9700 GPU and audio PCI IDs

---

8. Quick rollback (if needed)

To restore the original host behavior and let the R9700 be used by the host again:

# Restore GRUB command line (replace with your original line as needed)
sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="quiet"/' /etc/default/grub

# Remove the new config files
rm /etc/modules-load.d/vfio.conf
rm /etc/modprobe.d/blacklist-amdgpu-r9700.conf
rm /etc/modprobe.d/vfio.conf

# Regenerate and reboot
update-grub
update-initramfs -u -k all
reboot

After reboot, the R9700 will be bound by amdgpu and snd_hda_intel again.