Proxmox_GPU_Passthrough/README.md

# SOP: AMD Radeon AI Pro R9700 GPU Passthrough on Proxmox VE

**Repository:** `pch/Proxmox_GPU_Passthrough`  
**Node:** `mach1` (`192.168.5.11`)  
**GPU:** AMD Radeon AI Pro R9700 32GB (`1002:7551`)  
**GPU Audio Function:** AMD Navi 48 HDMI/DP Audio Controller (`1002:ab40`)  
**Date:** 2026-08-26  
**Goal:** Isolate the R9700 from the Proxmox VE host and bind it to `vfio-pci` so it can be passed through to a VM for AI/ML workloads.

---

## 1. Discover the GPU on the node

Log in to the Proxmox node via SSH and identify the GPU and its audio function.

```bash
ssh root@192.168.5.11
lspci -nn | grep -i 'AMD'
```

Expected output (excerpt):

```text
82:00.0 PCI bridge [0604]: AMD/ATI Navi 10 XL Upstream Port of PCI Express Switch [1002:1478]
83:00.0 PCI bridge [0604]: AMD/ATI Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
84:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
84:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
```

Key facts:

- GPU core is on PCI bus `0000:84:00.0`, vendor/device ID `1002:7551`.
- GPU audio is on PCI bus `0000:84:00.1`, vendor/device ID `1002:ab40`.
- Link speed: Gen5 x16 (`Speed 32GT/s, Width x16`).

---

## 2. Verify prerequisites

### 2.1 IOMMU is enabled

```bash
dmesg | grep -iE 'AMD-Vi|IOMMU'
```

Expected output:

```text
AMD-Vi: Extended features (0x58f77ef22294ade, 0x0): PPR X2APIC NX GT IA GA PC GA_vAPIC
AMD-Vi: Interrupt remapping enabled
AMD-Vi: X2APIC enabled
AMD-Vi: Virtual APIC enabled
```

### 2.2 CPU virtualization extensions are present

```bash
grep -iE 'svm|npt|avic' /proc/cpuinfo | head -2
```

Expected flags include `svm`, `npt`, `avic`.

### 2.3 IOMMU grouping is clean

```bash
for g in /sys/kernel/iommu_groups/*; do
  for d in $g/devices/*; do
    echo "$(basename $g): $(basename $d) $(lspci -nns $(basename $d))"
  done
done | grep -iE '1002:7551|1002:ab40'
```

Expected output:

```text
30: 0000:84:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
31: 0000:84:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
```

Both functions are in their **own IOMMU groups** (group `30` for the GPU, group `31` for audio). This means they can be passed through cleanly without needing the ACS override patch.

### 2.4 Check current driver binding

```bash
lspci -nnk | grep -iE '1002:7551|1002:ab40' -A 3
```

Before configuration, the output will look like:

```text
84:00.0 VGA compatible controller [0300]: ... Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
84:00.1 Audio device [0403]: ... Navi 48 HDMI/DP Audio Controller [1002:ab40]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
```

The GPU must be freed from both drivers before `vfio-pci` can bind.

---

## 3. Configure the host for VFIO passthrough

All commands below run as `root` on `mach1`.

### 3.1 Enable IOMMU passthrough mode in GRUB

Edit `/etc/default/grub`:

```bash
sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"/' /etc/default/grub
```

Resulting line:

```text
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"
```

- `amd_iommu=on`: Explicitly enables the AMD IOMMU (AMD-Vi) at boot.
- `iommu=pt`: Enables IOMMU **passthrough mode**, meaning DMA is only translated for devices assigned to VMs, reducing host overhead and avoiding conflicts between host drivers and the IOMMU.

### 3.2 Load VFIO modules at boot

Create `/etc/modules-load.d/vfio.conf`:

```bash
cat > /etc/modules-load.d/vfio.conf <<'EOF'
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
EOF
```

### 3.3 Blacklist the host GPU and audio drivers

Create `/etc/modprobe.d/blacklist-amdgpu-r9700.conf`:

```bash
cat > /etc/modprobe.d/blacklist-amdgpu-r9700.conf <<'EOF'
blacklist amdgpu
blacklist snd_hda_intel
EOF
```

Why blacklist both?

- `amdgpu` is the Linux kernel driver for AMD GPUs. If it loads, it will claim the R9700 and prevent `vfio-pci` from binding.
- `snd_hda_intel` is the generic High Definition Audio driver. It will claim the HDMI/DP audio function of the same AMD GPU (`84:00.1`). For clean passthrough, both functions must be free.

### 3.4 Bind VFIO to the specific PCI IDs

Create `/etc/modprobe.d/vfio.conf`:

```bash
cat > /etc/modprobe.d/vfio.conf <<'EOF'
options vfio-pci ids=1002:7551,1002:ab40
EOF
```

This approach is targeted: only the R9700 GPU and its audio function are captured by `vfio-pci`. Other AMD GPUs, if added later, are not affected.

### 3.5 Regenerate bootloader and initramfs

```bash
update-grub
update-initramfs -u -k all
```

---

## 4. Reboot and verify

### 4.1 Reboot the node

```bash
reboot
```

### 4.2 Verify kernel command line

```bash
cat /proc/cmdline
```

Expected:

```text
BOOT_IMAGE=/boot/vmlinuz-7.0.2-6-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt
```

### 4.3 Verify VFIO modules loaded

```bash
lsmod | grep -iE 'vfio|amdgpu'
```

Expected: `vfio_pci`, `vfio_pci_core`, `vfio_iommu_type1`, `vfio`, and `iommufd` present. `amdgpu` should **not** be loaded.

### 4.4 Verify device binding

```bash
lspci -nnk | grep -iE '1002:7551|1002:ab40' -A 3
```

Expected:

```text
84:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
        Subsystem: ASUSTeK Computer Inc. Device [1043:0626]
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu
84:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
```

> **Note:** The `Kernel modules:` line may still list `amdgpu` and `snd_hda_intel`. That is normal — it only means those drivers *could* drive the device. The important value is `Kernel driver in use: vfio-pci`.

### 4.5 Verify sysfs binding

```bash
ls /sys/bus/pci/drivers/vfio-pci/ | grep ':'
```

Expected:

```text
0000:84:00.0
0000:84:00.1
```

---

## 5. Proxmox VM configuration (when ready)

When you create the VM that will receive the R9700, use these settings:

| Setting | Value |
|--------|-------|
| **Machine type** | `q35` |
| **BIOS** | `OVMF (UEFI)` |
| **CPU type** | `host` |
| **CPU topology / NUMA** | Match host only if AI workload benefits; otherwise default |
| **Memory** | Static allocation recommended for AI workloads; disable ballooning |
| **Display** | Standard Proxmox console (VirtIO/VNC) — do **not** set R9700 as primary |
| **PCI passthrough device 1** | `0000:84:00.0` — Radeon AI PRO R9700 |
| **PCI passthrough device 2** | `0000:84:00.1` — HDMI/DP Audio |
| **PCI options** | Primary GPU: **No**; PCI-Express: **Yes**; ROM-Bar: **Yes** (default is usually fine) |

Inside the guest, install the appropriate AMD driver stack (e.g. ROCm on Linux, or Adrenalin/amdgpu driver on Windows) for AI workloads.

---

## 6. Lessons learned and important notes

### 6.1 Why both GRUB flags are needed

- `amd_iommu=on`: This is the on/off switch for AMD IOMMU. Without it, the kernel has no device isolation layer and cannot safely hand a PCIe device to a VM.
- `iommu=pt`: This enables **passthrough mode**. It tells the IOMMU to leave host DMA untranslated and only translate DMA for devices assigned to guests. This reduces host overhead and avoids IOMMU conflicts with the host.

Using only `amd_iommu=on` can sometimes work, but `iommu=pt` is the Proxmox-recommended setting for GPU passthrough hosts and avoids subtle bugs.

### 6.2 Why both `amdgpu` and `snd_hda_intel` must be blacklisted

A modern GPU is not one PCI device — it is multiple functions on the same card:

- `84:00.0`: the GPU itself (video/compute).
- `84:00.1`: the HDMI/DisplayPort audio controller.

The `amdgpu` kernel driver will bind to `84:00.0`. The `snd_hda_intel` driver — despite the name, this is the generic HDA audio driver used by AMD, Intel, and NVIDIA GPUs — will bind to `84:00.1`.

If either function remains claimed by the host, the passthrough will be incomplete or fail. Blacklisting both drivers ensures the entire card is free for `vfio-pci`.

### 6.3 Why the audio function matters even for AI-only workloads

Even if the VM will not output audio, passing through the audio function alongside the GPU is recommended because:

1. It prevents the host from retaining a claim on part of the card.
2. Some AMD GPU drivers expect both functions and behave more reliably when passed together.
3. The audio function is in its own clean IOMMU group (group `31`), so it adds no risk.

### 6.4 Use PCI IDs, not just a driver blacklist

The blacklist alone prevents the host drivers from loading, but it does not tell `vfio-pci` which devices to claim. The `options vfio-pci ids=1002:7551,1002:ab40` line explicitly binds the R9700 to VFIO. Using PCI IDs is more precise than a blanket blacklist and is safer if more GPUs are added later.

### 6.5 Why a reboot is mandatory

The `amdgpu` driver was already bound to the R9700 at the time of configuration. You cannot cleanly unbind a GPU driver that has initialized firmware, memory, and display engines without a reboot. A full reboot guarantees the new GRUB options, module load order, and blacklist take effect before any driver can claim the card.

### 6.6 BMC / host console video

`mach1` uses an ASpeed BMC VGA controller (`46:00.0`) for host console video. This means blacklisting `amdgpu` does **not** remove the host's ability to display video or log in locally. Always verify the host has an alternative console before blacklisting a GPU driver.

### 6.7 IOMMU groups are clean, no ACS override needed

The GPU is in IOMMU group `30` and the audio function is in group `31`. There are no other devices in either group. This is the ideal scenario for passthrough — no ACS override patch or risky kernel parameters are needed.

### 6.8 Do not create the VM as part of this SOP

This SOP intentionally stops at host preparation. VM creation, guest OS installation, and driver installation are separate steps and should be documented in their own procedure.

---

## 7. Files changed on `mach1`

| File | Purpose |
|------|---------|
| `/etc/default/grub` | Enables IOMMU and passthrough mode at boot |
| `/etc/modules-load.d/vfio.conf` | Loads VFIO framework modules at boot |
| `/etc/modprobe.d/blacklist-amdgpu-r9700.conf` | Prevents host drivers from claiming the R9700 |
| `/etc/modprobe.d/vfio.conf` | Binds `vfio-pci` to the R9700 GPU and audio PCI IDs |

---

## 8. Quick rollback (if needed)

To restore the original host behavior and let the R9700 be used by the host again:

```bash
# Restore GRUB command line
sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="quiet amdgpu.ppfeaturemask=0xffffffff"/' /etc/default/grub

# Remove the new config files
rm /etc/modules-load.d/vfio.conf
rm /etc/modprobe.d/blacklist-amdgpu-r9700.conf
rm /etc/modprobe.d/vfio.conf

# Regenerate and reboot
update-grub
update-initramfs -u -k all
reboot
```

After reboot, the R9700 will be bound by `amdgpu` and `snd_hda_intel` again.