From 4aae1fc252b499bd4a2dfb1c8792a4aca3ca675b Mon Sep 17 00:00:00 2001 From: PeterChrz Date: Wed, 26 Aug 2026 10:17:23 -0400 Subject: [PATCH] docs: add SOP for AMD Radeon AI Pro R9700 VFIO passthrough on mach1 --- README.md | 339 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 337 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 27cf742..c1104d2 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,338 @@ -# Proxmox_GPU_Passthrough +# SOP: AMD Radeon AI Pro R9700 GPU Passthrough on Proxmox VE -SOP and configuration for AMD AI Pro R9700 GPU passthrough on Proxmox VE (mach1) \ No newline at end of file +**Repository:** `pch/Proxmox_GPU_Passthrough` +**Node:** `mach1` (`192.168.5.11`) +**GPU:** AMD Radeon AI Pro R9700 32GB (`1002:7551`) +**GPU Audio Function:** AMD Navi 48 HDMI/DP Audio Controller (`1002:ab40`) +**Date:** 2026-08-26 +**Goal:** Isolate the R9700 from the Proxmox VE host and bind it to `vfio-pci` so it can be passed through to a VM for AI/ML workloads. + +--- + +## 1. Discover the GPU on the node + +Log in to the Proxmox node via SSH and identify the GPU and its audio function. + +```bash +ssh root@192.168.5.11 +lspci -nn | grep -i 'AMD' +``` + +Expected output (excerpt): + +```text +82:00.0 PCI bridge [0604]: AMD/ATI Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] +83:00.0 PCI bridge [0604]: AMD/ATI Navi 10 XL Downstream Port of PCI Express Switch [1002:1479] +84:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0) +84:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40] +``` + +Key facts: + +- GPU core is on PCI bus `0000:84:00.0`, vendor/device ID `1002:7551`. +- GPU audio is on PCI bus `0000:84:00.1`, vendor/device ID `1002:ab40`. +- Link speed: Gen5 x16 (`Speed 32GT/s, Width x16`). + +--- + +## 2. Verify prerequisites + +### 2.1 IOMMU is enabled + +```bash +dmesg | grep -iE 'AMD-Vi|IOMMU' +``` + +Expected output: + +```text +AMD-Vi: Extended features (0x58f77ef22294ade, 0x0): PPR X2APIC NX GT IA GA PC GA_vAPIC +AMD-Vi: Interrupt remapping enabled +AMD-Vi: X2APIC enabled +AMD-Vi: Virtual APIC enabled +``` + +### 2.2 CPU virtualization extensions are present + +```bash +grep -iE 'svm|npt|avic' /proc/cpuinfo | head -2 +``` + +Expected flags include `svm`, `npt`, `avic`. + +### 2.3 IOMMU grouping is clean + +```bash +for g in /sys/kernel/iommu_groups/*; do + for d in $g/devices/*; do + echo "$(basename $g): $(basename $d) $(lspci -nns $(basename $d))" + done +done | grep -iE '1002:7551|1002:ab40' +``` + +Expected output: + +```text +30: 0000:84:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0) +31: 0000:84:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40] +``` + +Both functions are in their **own IOMMU groups** (group `30` for the GPU, group `31` for audio). This means they can be passed through cleanly without needing the ACS override patch. + +### 2.4 Check current driver binding + +```bash +lspci -nnk | grep -iE '1002:7551|1002:ab40' -A 3 +``` + +Before configuration, the output will look like: + +```text +84:00.0 VGA compatible controller [0300]: ... Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0) + Kernel driver in use: amdgpu + Kernel modules: amdgpu +84:00.1 Audio device [0403]: ... Navi 48 HDMI/DP Audio Controller [1002:ab40] + Kernel driver in use: snd_hda_intel + Kernel modules: snd_hda_intel +``` + +The GPU must be freed from both drivers before `vfio-pci` can bind. + +--- + +## 3. Configure the host for VFIO passthrough + +All commands below run as `root` on `mach1`. + +### 3.1 Enable IOMMU passthrough mode in GRUB + +Edit `/etc/default/grub`: + +```bash +sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"/' /etc/default/grub +``` + +Resulting line: + +```text +GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt" +``` + +- `amd_iommu=on`: Explicitly enables the AMD IOMMU (AMD-Vi) at boot. +- `iommu=pt`: Enables IOMMU **passthrough mode**, meaning DMA is only translated for devices assigned to VMs, reducing host overhead and avoiding conflicts between host drivers and the IOMMU. + +### 3.2 Load VFIO modules at boot + +Create `/etc/modules-load.d/vfio.conf`: + +```bash +cat > /etc/modules-load.d/vfio.conf <<'EOF' +vfio +vfio_iommu_type1 +vfio_pci +vfio_virqfd +EOF +``` + +### 3.3 Blacklist the host GPU and audio drivers + +Create `/etc/modprobe.d/blacklist-amdgpu-r9700.conf`: + +```bash +cat > /etc/modprobe.d/blacklist-amdgpu-r9700.conf <<'EOF' +blacklist amdgpu +blacklist snd_hda_intel +EOF +``` + +Why blacklist both? + +- `amdgpu` is the Linux kernel driver for AMD GPUs. If it loads, it will claim the R9700 and prevent `vfio-pci` from binding. +- `snd_hda_intel` is the generic High Definition Audio driver. It will claim the HDMI/DP audio function of the same AMD GPU (`84:00.1`). For clean passthrough, both functions must be free. + +### 3.4 Bind VFIO to the specific PCI IDs + +Create `/etc/modprobe.d/vfio.conf`: + +```bash +cat > /etc/modprobe.d/vfio.conf <<'EOF' +options vfio-pci ids=1002:7551,1002:ab40 +EOF +``` + +This approach is targeted: only the R9700 GPU and its audio function are captured by `vfio-pci`. Other AMD GPUs, if added later, are not affected. + +### 3.5 Regenerate bootloader and initramfs + +```bash +update-grub +update-initramfs -u -k all +``` + +--- + +## 4. Reboot and verify + +### 4.1 Reboot the node + +```bash +reboot +``` + +### 4.2 Verify kernel command line + +```bash +cat /proc/cmdline +``` + +Expected: + +```text +BOOT_IMAGE=/boot/vmlinuz-7.0.2-6-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt +``` + +### 4.3 Verify VFIO modules loaded + +```bash +lsmod | grep -iE 'vfio|amdgpu' +``` + +Expected: `vfio_pci`, `vfio_pci_core`, `vfio_iommu_type1`, `vfio`, and `iommufd` present. `amdgpu` should **not** be loaded. + +### 4.4 Verify device binding + +```bash +lspci -nnk | grep -iE '1002:7551|1002:ab40' -A 3 +``` + +Expected: + +```text +84:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0) + Subsystem: ASUSTeK Computer Inc. Device [1043:0626] + Kernel driver in use: vfio-pci + Kernel modules: amdgpu +84:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40] + Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40] + Kernel driver in use: vfio-pci + Kernel modules: snd_hda_intel +``` + +> **Note:** The `Kernel modules:` line may still list `amdgpu` and `snd_hda_intel`. That is normal — it only means those drivers *could* drive the device. The important value is `Kernel driver in use: vfio-pci`. + +### 4.5 Verify sysfs binding + +```bash +ls /sys/bus/pci/drivers/vfio-pci/ | grep ':' +``` + +Expected: + +```text +0000:84:00.0 +0000:84:00.1 +``` + +--- + +## 5. Proxmox VM configuration (when ready) + +When you create the VM that will receive the R9700, use these settings: + +| Setting | Value | +|--------|-------| +| **Machine type** | `q35` | +| **BIOS** | `OVMF (UEFI)` | +| **CPU type** | `host` | +| **CPU topology / NUMA** | Match host only if AI workload benefits; otherwise default | +| **Memory** | Static allocation recommended for AI workloads; disable ballooning | +| **Display** | Standard Proxmox console (VirtIO/VNC) — do **not** set R9700 as primary | +| **PCI passthrough device 1** | `0000:84:00.0` — Radeon AI PRO R9700 | +| **PCI passthrough device 2** | `0000:84:00.1` — HDMI/DP Audio | +| **PCI options** | Primary GPU: **No**; PCI-Express: **Yes**; ROM-Bar: **Yes** (default is usually fine) | + +Inside the guest, install the appropriate AMD driver stack (e.g. ROCm on Linux, or Adrenalin/amdgpu driver on Windows) for AI workloads. + +--- + +## 6. Lessons learned and important notes + +### 6.1 Why both GRUB flags are needed + +- `amd_iommu=on`: This is the on/off switch for AMD IOMMU. Without it, the kernel has no device isolation layer and cannot safely hand a PCIe device to a VM. +- `iommu=pt`: This enables **passthrough mode**. It tells the IOMMU to leave host DMA untranslated and only translate DMA for devices assigned to guests. This reduces host overhead and avoids IOMMU conflicts with the host. + +Using only `amd_iommu=on` can sometimes work, but `iommu=pt` is the Proxmox-recommended setting for GPU passthrough hosts and avoids subtle bugs. + +### 6.2 Why both `amdgpu` and `snd_hda_intel` must be blacklisted + +A modern GPU is not one PCI device — it is multiple functions on the same card: + +- `84:00.0`: the GPU itself (video/compute). +- `84:00.1`: the HDMI/DisplayPort audio controller. + +The `amdgpu` kernel driver will bind to `84:00.0`. The `snd_hda_intel` driver — despite the name, this is the generic HDA audio driver used by AMD, Intel, and NVIDIA GPUs — will bind to `84:00.1`. + +If either function remains claimed by the host, the passthrough will be incomplete or fail. Blacklisting both drivers ensures the entire card is free for `vfio-pci`. + +### 6.3 Why the audio function matters even for AI-only workloads + +Even if the VM will not output audio, passing through the audio function alongside the GPU is recommended because: + +1. It prevents the host from retaining a claim on part of the card. +2. Some AMD GPU drivers expect both functions and behave more reliably when passed together. +3. The audio function is in its own clean IOMMU group (group `31`), so it adds no risk. + +### 6.4 Use PCI IDs, not just a driver blacklist + +The blacklist alone prevents the host drivers from loading, but it does not tell `vfio-pci` which devices to claim. The `options vfio-pci ids=1002:7551,1002:ab40` line explicitly binds the R9700 to VFIO. Using PCI IDs is more precise than a blanket blacklist and is safer if more GPUs are added later. + +### 6.5 Why a reboot is mandatory + +The `amdgpu` driver was already bound to the R9700 at the time of configuration. You cannot cleanly unbind a GPU driver that has initialized firmware, memory, and display engines without a reboot. A full reboot guarantees the new GRUB options, module load order, and blacklist take effect before any driver can claim the card. + +### 6.6 BMC / host console video + +`mach1` uses an ASpeed BMC VGA controller (`46:00.0`) for host console video. This means blacklisting `amdgpu` does **not** remove the host's ability to display video or log in locally. Always verify the host has an alternative console before blacklisting a GPU driver. + +### 6.7 IOMMU groups are clean, no ACS override needed + +The GPU is in IOMMU group `30` and the audio function is in group `31`. There are no other devices in either group. This is the ideal scenario for passthrough — no ACS override patch or risky kernel parameters are needed. + +### 6.8 Do not create the VM as part of this SOP + +This SOP intentionally stops at host preparation. VM creation, guest OS installation, and driver installation are separate steps and should be documented in their own procedure. + +--- + +## 7. Files changed on `mach1` + +| File | Purpose | +|------|---------| +| `/etc/default/grub` | Enables IOMMU and passthrough mode at boot | +| `/etc/modules-load.d/vfio.conf` | Loads VFIO framework modules at boot | +| `/etc/modprobe.d/blacklist-amdgpu-r9700.conf` | Prevents host drivers from claiming the R9700 | +| `/etc/modprobe.d/vfio.conf` | Binds `vfio-pci` to the R9700 GPU and audio PCI IDs | + +--- + +## 8. Quick rollback (if needed) + +To restore the original host behavior and let the R9700 be used by the host again: + +```bash +# Restore GRUB command line +sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="quiet amdgpu.ppfeaturemask=0xffffffff"/' /etc/default/grub + +# Remove the new config files +rm /etc/modules-load.d/vfio.conf +rm /etc/modprobe.d/blacklist-amdgpu-r9700.conf +rm /etc/modprobe.d/vfio.conf + +# Regenerate and reboot +update-grub +update-initramfs -u -k all +reboot +``` + +After reboot, the R9700 will be bound by `amdgpu` and `snd_hda_intel` again.