docs: add SOP for AMD Radeon AI Pro R9700 VFIO passthrough on mach1
This commit is contained in:
PeterChrz
parent
5fef045f11
commit
4aae1fc252
GPG key ID:
8F0826ECF7302C63
1 changed files with 337 additions and 2 deletions
339
README.md
339
README.md
|
|
@ -1,3 +1,338 @@
|
||||||
# Proxmox_GPU_Passthrough
|
# SOP: AMD Radeon AI Pro R9700 GPU Passthrough on Proxmox VE
|
||||||
|
|
||||||
SOP and configuration for AMD AI Pro R9700 GPU passthrough on Proxmox VE (mach1)
|
**Repository:** `pch/Proxmox_GPU_Passthrough`
|
||||||
|
**Node:** `mach1` (`192.168.5.11`)
|
||||||
|
**GPU:** AMD Radeon AI Pro R9700 32GB (`1002:7551`)
|
||||||
|
**GPU Audio Function:** AMD Navi 48 HDMI/DP Audio Controller (`1002:ab40`)
|
||||||
|
**Date:** 2026-08-26
|
||||||
|
**Goal:** Isolate the R9700 from the Proxmox VE host and bind it to `vfio-pci` so it can be passed through to a VM for AI/ML workloads.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Discover the GPU on the node
|
||||||
|
|
||||||
|
Log in to the Proxmox node via SSH and identify the GPU and its audio function.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh root@192.168.5.11
|
||||||
|
lspci -nn | grep -i 'AMD'
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected output (excerpt):
|
||||||
|
|
||||||
|
```text
|
||||||
|
82:00.0 PCI bridge [0604]: AMD/ATI Navi 10 XL Upstream Port of PCI Express Switch [1002:1478]
|
||||||
|
83:00.0 PCI bridge [0604]: AMD/ATI Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
|
||||||
|
84:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
|
||||||
|
84:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
|
||||||
|
```
|
||||||
|
|
||||||
|
Key facts:
|
||||||
|
|
||||||
|
- GPU core is on PCI bus `0000:84:00.0`, vendor/device ID `1002:7551`.
|
||||||
|
- GPU audio is on PCI bus `0000:84:00.1`, vendor/device ID `1002:ab40`.
|
||||||
|
- Link speed: Gen5 x16 (`Speed 32GT/s, Width x16`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Verify prerequisites
|
||||||
|
|
||||||
|
### 2.1 IOMMU is enabled
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dmesg | grep -iE 'AMD-Vi|IOMMU'
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected output:
|
||||||
|
|
||||||
|
```text
|
||||||
|
AMD-Vi: Extended features (0x58f77ef22294ade, 0x0): PPR X2APIC NX GT IA GA PC GA_vAPIC
|
||||||
|
AMD-Vi: Interrupt remapping enabled
|
||||||
|
AMD-Vi: X2APIC enabled
|
||||||
|
AMD-Vi: Virtual APIC enabled
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2.2 CPU virtualization extensions are present
|
||||||
|
|
||||||
|
```bash
|
||||||
|
grep -iE 'svm|npt|avic' /proc/cpuinfo | head -2
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected flags include `svm`, `npt`, `avic`.
|
||||||
|
|
||||||
|
### 2.3 IOMMU grouping is clean
|
||||||
|
|
||||||
|
```bash
|
||||||
|
for g in /sys/kernel/iommu_groups/*; do
|
||||||
|
for d in $g/devices/*; do
|
||||||
|
echo "$(basename $g): $(basename $d) $(lspci -nns $(basename $d))"
|
||||||
|
done
|
||||||
|
done | grep -iE '1002:7551|1002:ab40'
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected output:
|
||||||
|
|
||||||
|
```text
|
||||||
|
30: 0000:84:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
|
||||||
|
31: 0000:84:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
|
||||||
|
```
|
||||||
|
|
||||||
|
Both functions are in their **own IOMMU groups** (group `30` for the GPU, group `31` for audio). This means they can be passed through cleanly without needing the ACS override patch.
|
||||||
|
|
||||||
|
### 2.4 Check current driver binding
|
||||||
|
|
||||||
|
```bash
|
||||||
|
lspci -nnk | grep -iE '1002:7551|1002:ab40' -A 3
|
||||||
|
```
|
||||||
|
|
||||||
|
Before configuration, the output will look like:
|
||||||
|
|
||||||
|
```text
|
||||||
|
84:00.0 VGA compatible controller [0300]: ... Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
|
||||||
|
Kernel driver in use: amdgpu
|
||||||
|
Kernel modules: amdgpu
|
||||||
|
84:00.1 Audio device [0403]: ... Navi 48 HDMI/DP Audio Controller [1002:ab40]
|
||||||
|
Kernel driver in use: snd_hda_intel
|
||||||
|
Kernel modules: snd_hda_intel
|
||||||
|
```
|
||||||
|
|
||||||
|
The GPU must be freed from both drivers before `vfio-pci` can bind.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Configure the host for VFIO passthrough
|
||||||
|
|
||||||
|
All commands below run as `root` on `mach1`.
|
||||||
|
|
||||||
|
### 3.1 Enable IOMMU passthrough mode in GRUB
|
||||||
|
|
||||||
|
Edit `/etc/default/grub`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"/' /etc/default/grub
|
||||||
|
```
|
||||||
|
|
||||||
|
Resulting line:
|
||||||
|
|
||||||
|
```text
|
||||||
|
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"
|
||||||
|
```
|
||||||
|
|
||||||
|
- `amd_iommu=on`: Explicitly enables the AMD IOMMU (AMD-Vi) at boot.
|
||||||
|
- `iommu=pt`: Enables IOMMU **passthrough mode**, meaning DMA is only translated for devices assigned to VMs, reducing host overhead and avoiding conflicts between host drivers and the IOMMU.
|
||||||
|
|
||||||
|
### 3.2 Load VFIO modules at boot
|
||||||
|
|
||||||
|
Create `/etc/modules-load.d/vfio.conf`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cat > /etc/modules-load.d/vfio.conf <<'EOF'
|
||||||
|
vfio
|
||||||
|
vfio_iommu_type1
|
||||||
|
vfio_pci
|
||||||
|
vfio_virqfd
|
||||||
|
EOF
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.3 Blacklist the host GPU and audio drivers
|
||||||
|
|
||||||
|
Create `/etc/modprobe.d/blacklist-amdgpu-r9700.conf`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cat > /etc/modprobe.d/blacklist-amdgpu-r9700.conf <<'EOF'
|
||||||
|
blacklist amdgpu
|
||||||
|
blacklist snd_hda_intel
|
||||||
|
EOF
|
||||||
|
```
|
||||||
|
|
||||||
|
Why blacklist both?
|
||||||
|
|
||||||
|
- `amdgpu` is the Linux kernel driver for AMD GPUs. If it loads, it will claim the R9700 and prevent `vfio-pci` from binding.
|
||||||
|
- `snd_hda_intel` is the generic High Definition Audio driver. It will claim the HDMI/DP audio function of the same AMD GPU (`84:00.1`). For clean passthrough, both functions must be free.
|
||||||
|
|
||||||
|
### 3.4 Bind VFIO to the specific PCI IDs
|
||||||
|
|
||||||
|
Create `/etc/modprobe.d/vfio.conf`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cat > /etc/modprobe.d/vfio.conf <<'EOF'
|
||||||
|
options vfio-pci ids=1002:7551,1002:ab40
|
||||||
|
EOF
|
||||||
|
```
|
||||||
|
|
||||||
|
This approach is targeted: only the R9700 GPU and its audio function are captured by `vfio-pci`. Other AMD GPUs, if added later, are not affected.
|
||||||
|
|
||||||
|
### 3.5 Regenerate bootloader and initramfs
|
||||||
|
|
||||||
|
```bash
|
||||||
|
update-grub
|
||||||
|
update-initramfs -u -k all
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Reboot and verify
|
||||||
|
|
||||||
|
### 4.1 Reboot the node
|
||||||
|
|
||||||
|
```bash
|
||||||
|
reboot
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.2 Verify kernel command line
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cat /proc/cmdline
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected:
|
||||||
|
|
||||||
|
```text
|
||||||
|
BOOT_IMAGE=/boot/vmlinuz-7.0.2-6-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.3 Verify VFIO modules loaded
|
||||||
|
|
||||||
|
```bash
|
||||||
|
lsmod | grep -iE 'vfio|amdgpu'
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: `vfio_pci`, `vfio_pci_core`, `vfio_iommu_type1`, `vfio`, and `iommufd` present. `amdgpu` should **not** be loaded.
|
||||||
|
|
||||||
|
### 4.4 Verify device binding
|
||||||
|
|
||||||
|
```bash
|
||||||
|
lspci -nnk | grep -iE '1002:7551|1002:ab40' -A 3
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected:
|
||||||
|
|
||||||
|
```text
|
||||||
|
84:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
|
||||||
|
Subsystem: ASUSTeK Computer Inc. Device [1043:0626]
|
||||||
|
Kernel driver in use: vfio-pci
|
||||||
|
Kernel modules: amdgpu
|
||||||
|
84:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
|
||||||
|
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
|
||||||
|
Kernel driver in use: vfio-pci
|
||||||
|
Kernel modules: snd_hda_intel
|
||||||
|
```
|
||||||
|
|
||||||
|
> **Note:** The `Kernel modules:` line may still list `amdgpu` and `snd_hda_intel`. That is normal — it only means those drivers *could* drive the device. The important value is `Kernel driver in use: vfio-pci`.
|
||||||
|
|
||||||
|
### 4.5 Verify sysfs binding
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ls /sys/bus/pci/drivers/vfio-pci/ | grep ':'
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected:
|
||||||
|
|
||||||
|
```text
|
||||||
|
0000:84:00.0
|
||||||
|
0000:84:00.1
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Proxmox VM configuration (when ready)
|
||||||
|
|
||||||
|
When you create the VM that will receive the R9700, use these settings:
|
||||||
|
|
||||||
|
| Setting | Value |
|
||||||
|
|--------|-------|
|
||||||
|
| **Machine type** | `q35` |
|
||||||
|
| **BIOS** | `OVMF (UEFI)` |
|
||||||
|
| **CPU type** | `host` |
|
||||||
|
| **CPU topology / NUMA** | Match host only if AI workload benefits; otherwise default |
|
||||||
|
| **Memory** | Static allocation recommended for AI workloads; disable ballooning |
|
||||||
|
| **Display** | Standard Proxmox console (VirtIO/VNC) — do **not** set R9700 as primary |
|
||||||
|
| **PCI passthrough device 1** | `0000:84:00.0` — Radeon AI PRO R9700 |
|
||||||
|
| **PCI passthrough device 2** | `0000:84:00.1` — HDMI/DP Audio |
|
||||||
|
| **PCI options** | Primary GPU: **No**; PCI-Express: **Yes**; ROM-Bar: **Yes** (default is usually fine) |
|
||||||
|
|
||||||
|
Inside the guest, install the appropriate AMD driver stack (e.g. ROCm on Linux, or Adrenalin/amdgpu driver on Windows) for AI workloads.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Lessons learned and important notes
|
||||||
|
|
||||||
|
### 6.1 Why both GRUB flags are needed
|
||||||
|
|
||||||
|
- `amd_iommu=on`: This is the on/off switch for AMD IOMMU. Without it, the kernel has no device isolation layer and cannot safely hand a PCIe device to a VM.
|
||||||
|
- `iommu=pt`: This enables **passthrough mode**. It tells the IOMMU to leave host DMA untranslated and only translate DMA for devices assigned to guests. This reduces host overhead and avoids IOMMU conflicts with the host.
|
||||||
|
|
||||||
|
Using only `amd_iommu=on` can sometimes work, but `iommu=pt` is the Proxmox-recommended setting for GPU passthrough hosts and avoids subtle bugs.
|
||||||
|
|
||||||
|
### 6.2 Why both `amdgpu` and `snd_hda_intel` must be blacklisted
|
||||||
|
|
||||||
|
A modern GPU is not one PCI device — it is multiple functions on the same card:
|
||||||
|
|
||||||
|
- `84:00.0`: the GPU itself (video/compute).
|
||||||
|
- `84:00.1`: the HDMI/DisplayPort audio controller.
|
||||||
|
|
||||||
|
The `amdgpu` kernel driver will bind to `84:00.0`. The `snd_hda_intel` driver — despite the name, this is the generic HDA audio driver used by AMD, Intel, and NVIDIA GPUs — will bind to `84:00.1`.
|
||||||
|
|
||||||
|
If either function remains claimed by the host, the passthrough will be incomplete or fail. Blacklisting both drivers ensures the entire card is free for `vfio-pci`.
|
||||||
|
|
||||||
|
### 6.3 Why the audio function matters even for AI-only workloads
|
||||||
|
|
||||||
|
Even if the VM will not output audio, passing through the audio function alongside the GPU is recommended because:
|
||||||
|
|
||||||
|
1. It prevents the host from retaining a claim on part of the card.
|
||||||
|
2. Some AMD GPU drivers expect both functions and behave more reliably when passed together.
|
||||||
|
3. The audio function is in its own clean IOMMU group (group `31`), so it adds no risk.
|
||||||
|
|
||||||
|
### 6.4 Use PCI IDs, not just a driver blacklist
|
||||||
|
|
||||||
|
The blacklist alone prevents the host drivers from loading, but it does not tell `vfio-pci` which devices to claim. The `options vfio-pci ids=1002:7551,1002:ab40` line explicitly binds the R9700 to VFIO. Using PCI IDs is more precise than a blanket blacklist and is safer if more GPUs are added later.
|
||||||
|
|
||||||
|
### 6.5 Why a reboot is mandatory
|
||||||
|
|
||||||
|
The `amdgpu` driver was already bound to the R9700 at the time of configuration. You cannot cleanly unbind a GPU driver that has initialized firmware, memory, and display engines without a reboot. A full reboot guarantees the new GRUB options, module load order, and blacklist take effect before any driver can claim the card.
|
||||||
|
|
||||||
|
### 6.6 BMC / host console video
|
||||||
|
|
||||||
|
`mach1` uses an ASpeed BMC VGA controller (`46:00.0`) for host console video. This means blacklisting `amdgpu` does **not** remove the host's ability to display video or log in locally. Always verify the host has an alternative console before blacklisting a GPU driver.
|
||||||
|
|
||||||
|
### 6.7 IOMMU groups are clean, no ACS override needed
|
||||||
|
|
||||||
|
The GPU is in IOMMU group `30` and the audio function is in group `31`. There are no other devices in either group. This is the ideal scenario for passthrough — no ACS override patch or risky kernel parameters are needed.
|
||||||
|
|
||||||
|
### 6.8 Do not create the VM as part of this SOP
|
||||||
|
|
||||||
|
This SOP intentionally stops at host preparation. VM creation, guest OS installation, and driver installation are separate steps and should be documented in their own procedure.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Files changed on `mach1`
|
||||||
|
|
||||||
|
| File | Purpose |
|
||||||
|
|------|---------|
|
||||||
|
| `/etc/default/grub` | Enables IOMMU and passthrough mode at boot |
|
||||||
|
| `/etc/modules-load.d/vfio.conf` | Loads VFIO framework modules at boot |
|
||||||
|
| `/etc/modprobe.d/blacklist-amdgpu-r9700.conf` | Prevents host drivers from claiming the R9700 |
|
||||||
|
| `/etc/modprobe.d/vfio.conf` | Binds `vfio-pci` to the R9700 GPU and audio PCI IDs |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Quick rollback (if needed)
|
||||||
|
|
||||||
|
To restore the original host behavior and let the R9700 be used by the host again:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Restore GRUB command line
|
||||||
|
sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="quiet amdgpu.ppfeaturemask=0xffffffff"/' /etc/default/grub
|
||||||
|
|
||||||
|
# Remove the new config files
|
||||||
|
rm /etc/modules-load.d/vfio.conf
|
||||||
|
rm /etc/modprobe.d/blacklist-amdgpu-r9700.conf
|
||||||
|
rm /etc/modprobe.d/vfio.conf
|
||||||
|
|
||||||
|
# Regenerate and reboot
|
||||||
|
update-grub
|
||||||
|
update-initramfs -u -k all
|
||||||
|
reboot
|
||||||
|
```
|
||||||
|
|
||||||
|
After reboot, the R9700 will be bound by `amdgpu` and `snd_hda_intel` again.
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue