# SOP: AMD Radeon AI Pro R9700 GPU Passthrough on Proxmox VE

**Repository:** `pch/Proxmox_GPU_Passthrough`  
**Target:** A Proxmox VE compute node dedicated to AI/ML GPU passthrough  
**GPU:** AMD Radeon AI Pro R9700 32GB (`1002:7551`)  
**GPU Audio Function:** AMD Navi 48 HDMI/DP Audio Controller (`1002:ab40`)  
**Date:** 2026-08-26  
**Goal:** Isolate the R9700 from the Proxmox VE host and bind it to `vfio-pci` so it can be passed through to a VM for AI/ML workloads.

---

## 1. Discover the GPU on the node

Log in to the Proxmox node via SSH and identify the GPU and its audio function.

```bash
ssh root@<proxmox node>
lspci -nn | grep -i 'AMD'
```

Expected output (excerpt — values will vary by system):

```text
<pp>:00.0 PCI bridge [0604]: AMD/ATI Navi 10 XL Upstream Port of PCI Express Switch [1002:1478]
<pp>:00.0 PCI bridge [0604]: AMD/ATI Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
<pp>:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
<pp>:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
```

Where `<pp>` is the PCI domain/bus/slot address assigned by the host (e.g. `84:00`).

Key facts:

- GPU core vendor/device ID: `1002:7551`.
- GPU audio vendor/device ID: `1002:ab40`.
- Link speed should be Gen5 x16 (`Speed 32GT/s, Width x16`).

---

## 2. Verify prerequisites

### 2.1 IOMMU is enabled

```bash
dmesg | grep -iE 'AMD-Vi|IOMMU'
```

Expected output:

```text
AMD-Vi: Extended features (...): PPR X2APIC NX GT IA GA PC GA_vAPIC
AMD-Vi: Interrupt remapping enabled
AMD-Vi: X2APIC enabled
AMD-Vi: Virtual APIC enabled
```

### 2.2 CPU virtualization extensions are present

```bash
grep -iE 'svm|npt|avic' /proc/cpuinfo | head -2
```

Expected flags include `svm`, `npt`, `avic`.

### 2.3 IOMMU grouping is clean

```bash
for g in /sys/kernel/iommu_groups/*; do
  for d in $g/devices/*; do
    echo "$(basename $g): $(basename $d) $(lspci -nns $(basename $d))"
  done
done | grep -iE '1002:7551|1002:ab40'
```

Expected output (group numbers and addresses will vary):

```text
<gpu group>: 0000:<pp>:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
<audio group>: 0000:<pp>:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
```

Both functions should be in their **own IOMMU groups**. If they are the only devices in their respective groups, they can be passed through cleanly without needing the ACS override patch.

### 2.4 Check current driver binding

```bash
lspci -nnk | grep -iE '1002:7551|1002:ab40' -A 3
```

Before configuration, the output will look like:

```text
<pp>:00.0 VGA compatible controller [0300]: ... Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
<pp>:00.1 Audio device [0403]: ... Navi 48 HDMI/DP Audio Controller [1002:ab40]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
```

The GPU must be freed from both drivers before `vfio-pci` can bind.

---

## 3. Configure the host for VFIO passthrough

All commands below run as `root` on the Proxmox node.

### 3.1 Enable IOMMU passthrough mode in GRUB

Edit `/etc/default/grub`:

```bash
sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"/' /etc/default/grub
```

Resulting line:

```text
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"
```

- `amd_iommu=on`: Explicitly enables the AMD IOMMU (AMD-Vi) at boot.
- `iommu=pt`: Enables IOMMU **passthrough mode**, meaning DMA is only translated for devices assigned to VMs, reducing host overhead and avoiding conflicts between host drivers and the IOMMU.

### 3.2 Load VFIO modules at boot

Create `/etc/modules-load.d/vfio.conf`:

```bash
cat > /etc/modules-load.d/vfio.conf <<'EOF'
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
EOF
```

### 3.3 Blacklist the host GPU and audio drivers

Create `/etc/modprobe.d/blacklist-amdgpu-r9700.conf`:

```bash
cat > /etc/modprobe.d/blacklist-amdgpu-r9700.conf <<'EOF'
blacklist amdgpu
blacklist snd_hda_intel
EOF
```

Why blacklist both?

- `amdgpu` is the Linux kernel driver for AMD GPUs. If it loads, it will claim the R9700 and prevent `vfio-pci` from binding.
- `snd_hda_intel` is the generic High Definition Audio driver. It will claim the HDMI/DP audio function of the same AMD GPU (`<pp>:00.1`). For clean passthrough, both functions must be free.

### 3.4 Bind VFIO to the specific PCI IDs

Create `/etc/modprobe.d/vfio.conf`:

```bash
cat > /etc/modprobe.d/vfio.conf <<'EOF'
options vfio-pci ids=1002:7551,1002:ab40
EOF
```

This approach is targeted: only the R9700 GPU and its audio function are captured by `vfio-pci`. Other AMD GPUs, if added later, are not affected.

### 3.5 Regenerate bootloader and initramfs

```bash
update-grub
update-initramfs -u -k all
```

---

## 4. Reboot and verify

### 4.1 Reboot the node

```bash
reboot
```

### 4.2 Verify kernel command line

```bash
cat /proc/cmdline
```

Expected:

```text
BOOT_IMAGE=/boot/vmlinuz-... root=... ro quiet amd_iommu=on iommu=pt
```

### 4.3 Verify VFIO modules loaded

```bash
lsmod | grep -iE 'vfio|amdgpu'
```

Expected: `vfio_pci`, `vfio_pci_core`, `vfio_iommu_type1`, `vfio`, and `iommufd` present. `amdgpu` should **not** be loaded.

### 4.4 Verify device binding

```bash
lspci -nnk | grep -iE '1002:7551|1002:ab40' -A 3
```

Expected:

```text
<pp>:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu
<pp>:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
```

> **Note:** The `Kernel modules:` line may still list `amdgpu` and `snd_hda_intel`. That is normal — it only means those drivers *could* drive the device. The important value is `Kernel driver in use: vfio-pci`.

### 4.5 Verify sysfs binding

```bash
ls /sys/bus/pci/drivers/vfio-pci/ | grep ':'
```

Expected:

```text
0000:<pp>:00.0
0000:<pp>:00.1
```

---

## 5. Proxmox VM configuration (when ready)

When you create the VM that will receive the R9700, use these settings:

| Setting | Value |
|--------|-------|
| **Machine type** | `q35` |
| **BIOS** | `OVMF (UEFI)` |
| **CPU type** | `host` |
| **CPU topology / NUMA** | Match host only if AI workload benefits; otherwise default |
| **Memory** | Static allocation recommended for AI workloads; disable ballooning |
| **Display** | Standard Proxmox console (VirtIO/VNC) — do **not** set R9700 as primary |
| **PCI passthrough device 1** | `0000:<pp>:00.0` — Radeon AI PRO R9700 |
| **PCI passthrough device 2** | `0000:<pp>:00.1` — HDMI/DP Audio |
| **PCI options** | Primary GPU: **No**; PCI-Express: **Yes**; ROM-Bar: **Yes** (default is usually fine) |

Inside the guest, install the appropriate AMD driver stack (e.g. ROCm on Linux, or Adrenalin/amdgpu driver on Windows) for AI workloads.

---

## 6. Lessons learned and important notes

### 6.1 Why both GRUB flags are needed

- `amd_iommu=on`: This is the on/off switch for AMD IOMMU. Without it, the kernel has no device isolation layer and cannot safely hand a PCIe device to a VM.
- `iommu=pt`: This enables **passthrough mode**. It tells the IOMMU to leave host DMA untranslated and only translate DMA for devices assigned to guests. This reduces host overhead and avoids IOMMU conflicts with the host.

Using only `amd_iommu=on` can sometimes work, but `iommu=pt` is the Proxmox-recommended setting for GPU passthrough hosts and avoids subtle bugs.

### 6.2 Why both `amdgpu` and `snd_hda_intel` must be blacklisted

A modern GPU is not one PCI device — it is multiple functions on the same card:

- `<pp>:00.0`: the GPU itself (video/compute).
- `<pp>:00.1`: the HDMI/DisplayPort audio controller.

The `amdgpu` kernel driver will bind to `<pp>:00.0`. The `snd_hda_intel` driver — despite the name, this is the generic HDA audio driver used by AMD, Intel, and NVIDIA GPUs — will bind to `<pp>:00.1`.

If either function remains claimed by the host, the passthrough will be incomplete or fail. Blacklisting both drivers ensures the entire card is free for `vfio-pci`.

### 6.3 Why the audio function matters even for AI-only workloads

Even if the VM will not output audio, passing through the audio function alongside the GPU is recommended because:

1. It prevents the host from retaining a claim on part of the card.
2. Some AMD GPU drivers expect both functions and behave more reliably when passed together.
3. The audio function is in its own clean IOMMU group, so it adds no risk.

### 6.4 Use PCI IDs, not just a driver blacklist

The blacklist alone prevents the host drivers from loading, but it does not tell `vfio-pci` which devices to claim. The `options vfio-pci ids=1002:7551,1002:ab40` line explicitly binds the R9700 to VFIO. Using PCI IDs is more precise than a blanket blacklist and is safer if more GPUs are added later.

### 6.5 Why a reboot is mandatory

The `amdgpu` driver was already bound to the R9700 at the time of configuration. You cannot cleanly unbind a GPU driver that has initialized firmware, memory, and display engines without a reboot. A full reboot guarantees the new GRUB options, module load order, and blacklist take effect before any driver can claim the card.

### 6.6 BMC / host console video

Always verify the host has an alternative console (e.g. an ASpeed BMC VGA controller, a separate low-end GPU, or serial console) before blacklisting the primary GPU driver. Blacklisting `amdgpu` must not remove the host's ability to display video or log in locally.

### 6.7 IOMMU groups are clean, no ACS override needed

For this setup, the GPU was the only device in its IOMMU group and the audio function was the only device in its group. This is the ideal scenario for passthrough — no ACS override patch or risky kernel parameters are needed. If your system groups the GPU with other devices, additional steps may be required.

### 6.8 Do not create the VM as part of this SOP

This SOP intentionally stops at host preparation. VM creation, guest OS installation, and driver installation are separate steps and should be documented in their own procedure.

---

## 7. Files changed on the Proxmox node

| File | Purpose |
|------|---------|
| `/etc/default/grub` | Enables IOMMU and passthrough mode at boot |
| `/etc/modules-load.d/vfio.conf` | Loads VFIO framework modules at boot |
| `/etc/modprobe.d/blacklist-amdgpu-r9700.conf` | Prevents host drivers from claiming the R9700 |
| `/etc/modprobe.d/vfio.conf` | Binds `vfio-pci` to the R9700 GPU and audio PCI IDs |

---

## 8. Quick rollback (if needed)

To restore the original host behavior and let the R9700 be used by the host again:

```bash
# Restore GRUB command line (replace with your original line as needed)
sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="quiet"/' /etc/default/grub

# Remove the new config files
rm /etc/modules-load.d/vfio.conf
rm /etc/modprobe.d/blacklist-amdgpu-r9700.conf
rm /etc/modprobe.d/vfio.conf

# Regenerate and reboot
update-grub
update-initramfs -u -k all
reboot
```

After reboot, the R9700 will be bound by `amdgpu` and `snd_hda_intel` again.