pch/Proxmox_GPU_Passthrough
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: sanitize SOP by removing node name, IP, and PCI addresses

Browse source
This commit is contained in:
PeterChrz 2026-08-26 10:19:30 -04:00
parent 4aae1fc252
commit 8d5dc4f3e0
Signed by untrusted user who does not match committer: pch
GPG key ID: 8F0826ECF7302C63
1 changed files with 37 additions and 37 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

74
README.md
View file

@ -1,7 +1,7 @@
# SOP: AMD Radeon AI Pro R9700 GPU Passthrough on Proxmox VE
**Repository:** `pch/Proxmox_GPU_Passthrough`
**Node:** `mach1` (`192.168.5.11`)
**Target:** A Proxmox VE compute node dedicated to AI/ML GPU passthrough
**GPU:** AMD Radeon AI Pro R9700 32GB (`1002:7551`)
**GPU Audio Function:** AMD Navi 48 HDMI/DP Audio Controller (`1002:ab40`)
**Date:** 2026-08-26
@ -14,24 +14,26 @@
Log in to the Proxmox node via SSH and identify the GPU and its audio function.
```bash
ssh root@192.168.5.11
ssh root@<proxmox node>
lspci -nn | grep -i 'AMD'
```
Expected output (excerpt):
Expected output (excerpt — values will vary by system):
```text
82:00.0 PCI bridge [0604]: AMD/ATI Navi 10 XL Upstream Port of PCI Express Switch [1002:1478]
83:00.0 PCI bridge [0604]: AMD/ATI Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
84:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
84:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
<pp>:00.0 PCI bridge [0604]: AMD/ATI Navi 10 XL Upstream Port of PCI Express Switch [1002:1478]
<pp>:00.0 PCI bridge [0604]: AMD/ATI Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
<pp>:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
<pp>:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
```
Where `<pp>` is the PCI domain/bus/slot address assigned by the host (e.g. `84:00`).
Key facts:
- GPU core is on PCI bus `0000:84:00.0`, vendor/device ID `1002:7551`.
- GPU audio is on PCI bus `0000:84:00.1`, vendor/device ID `1002:ab40`.
- Link speed: Gen5 x16 (`Speed 32GT/s, Width x16`).
- GPU core vendor/device ID: `1002:7551`.
- GPU audio vendor/device ID: `1002:ab40`.
- Link speed should be Gen5 x16 (`Speed 32GT/s, Width x16`).
---
@ -46,7 +48,7 @@ dmesg | grep -iE 'AMD-Vi|IOMMU'
Expected output:
```text
AMD-Vi: Extended features (0x58f77ef22294ade, 0x0): PPR X2APIC NX GT IA GA PC GA_vAPIC
AMD-Vi: Extended features (...): PPR X2APIC NX GT IA GA PC GA_vAPIC
AMD-Vi: Interrupt remapping enabled
AMD-Vi: X2APIC enabled
AMD-Vi: Virtual APIC enabled
@ -70,14 +72,14 @@ for g in /sys/kernel/iommu_groups/*; do
done | grep -iE '1002:7551|1002:ab40'
```
Expected output:
Expected output (group numbers and addresses will vary):
```text
30: 0000:84:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
31: 0000:84:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
<gpu group>: 0000:<pp>:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
<audio group>: 0000:<pp>:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
```
Both functions are in their **own IOMMU groups** (group `30` for the GPU, group `31` for audio). This means they can be passed through cleanly without needing the ACS override patch.
Both functions should be in their **own IOMMU groups**. If they are the only devices in their respective groups, they can be passed through cleanly without needing the ACS override patch.
### 2.4 Check current driver binding
@ -88,10 +90,10 @@ lspci -nnk | grep -iE '1002:7551|1002:ab40' -A 3
Before configuration, the output will look like:
```text
84:00.0 VGA compatible controller [0300]: ... Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
<pp>:00.0 VGA compatible controller [0300]: ... Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
Kernel driver in use: amdgpu
Kernel modules: amdgpu
84:00.1 Audio device [0403]: ... Navi 48 HDMI/DP Audio Controller [1002:ab40]
<pp>:00.1 Audio device [0403]: ... Navi 48 HDMI/DP Audio Controller [1002:ab40]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
```
@ -102,7 +104,7 @@ The GPU must be freed from both drivers before `vfio-pci` can bind.
## 3. Configure the host for VFIO passthrough
All commands below run as `root` on `mach1`.
All commands below run as `root` on the Proxmox node.
### 3.1 Enable IOMMU passthrough mode in GRUB
@ -148,7 +150,7 @@ EOF
Why blacklist both?
- `amdgpu` is the Linux kernel driver for AMD GPUs. If it loads, it will claim the R9700 and prevent `vfio-pci` from binding.
- `snd_hda_intel` is the generic High Definition Audio driver. It will claim the HDMI/DP audio function of the same AMD GPU (`84:00.1`). For clean passthrough, both functions must be free.
- `snd_hda_intel` is the generic High Definition Audio driver. It will claim the HDMI/DP audio function of the same AMD GPU (`<pp>:00.1`). For clean passthrough, both functions must be free.
### 3.4 Bind VFIO to the specific PCI IDs
@ -188,7 +190,7 @@ cat /proc/cmdline
Expected:
```text
BOOT_IMAGE=/boot/vmlinuz-7.0.2-6-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt
BOOT_IMAGE=/boot/vmlinuz-... root=... ro quiet amd_iommu=on iommu=pt
```
### 4.3 Verify VFIO modules loaded
@ -208,12 +210,10 @@ lspci -nnk | grep -iE '1002:7551|1002:ab40' -A 3
Expected:
```text
84:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
Subsystem: ASUSTeK Computer Inc. Device [1043:0626]
<pp>:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551] (rev c0)
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
84:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
<pp>:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller [1002:ab40]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
```
@ -229,8 +229,8 @@ ls /sys/bus/pci/drivers/vfio-pci/ | grep ':'
Expected:
```text
0000:84:00.0
0000:84:00.1
0000:<pp>:00.0
0000:<pp>:00.1
```
---
@ -247,8 +247,8 @@ When you create the VM that will receive the R9700, use these settings:
| **CPU topology / NUMA** | Match host only if AI workload benefits; otherwise default |
| **Memory** | Static allocation recommended for AI workloads; disable ballooning |
| **Display** | Standard Proxmox console (VirtIO/VNC) — do **not** set R9700 as primary |
| **PCI passthrough device 1** | `0000:84:00.0` — Radeon AI PRO R9700 |
| **PCI passthrough device 2** | `0000:84:00.1` — HDMI/DP Audio |
| **PCI passthrough device 1** | `0000:<pp>:00.0` — Radeon AI PRO R9700 |
| **PCI passthrough device 2** | `0000:<pp>:00.1` — HDMI/DP Audio |
| **PCI options** | Primary GPU: **No**; PCI-Express: **Yes**; ROM-Bar: **Yes** (default is usually fine) |
Inside the guest, install the appropriate AMD driver stack (e.g. ROCm on Linux, or Adrenalin/amdgpu driver on Windows) for AI workloads.
@ -268,10 +268,10 @@ Using only `amd_iommu=on` can sometimes work, but `iommu=pt` is the Proxmox-reco
A modern GPU is not one PCI device — it is multiple functions on the same card:
- `84:00.0`: the GPU itself (video/compute).
- `84:00.1`: the HDMI/DisplayPort audio controller.
- `<pp>:00.0`: the GPU itself (video/compute).
- `<pp>:00.1`: the HDMI/DisplayPort audio controller.
The `amdgpu` kernel driver will bind to `84:00.0`. The `snd_hda_intel` driver — despite the name, this is the generic HDA audio driver used by AMD, Intel, and NVIDIA GPUs — will bind to `84:00.1`.
The `amdgpu` kernel driver will bind to `<pp>:00.0`. The `snd_hda_intel` driver — despite the name, this is the generic HDA audio driver used by AMD, Intel, and NVIDIA GPUs — will bind to `<pp>:00.1`.
If either function remains claimed by the host, the passthrough will be incomplete or fail. Blacklisting both drivers ensures the entire card is free for `vfio-pci`.
@ -281,7 +281,7 @@ Even if the VM will not output audio, passing through the audio function alongsi
1. It prevents the host from retaining a claim on part of the card.
2. Some AMD GPU drivers expect both functions and behave more reliably when passed together.
3. The audio function is in its own clean IOMMU group (group `31`), so it adds no risk.
3. The audio function is in its own clean IOMMU group, so it adds no risk.
### 6.4 Use PCI IDs, not just a driver blacklist
@ -293,11 +293,11 @@ The `amdgpu` driver was already bound to the R9700 at the time of configuration.
### 6.6 BMC / host console video
`mach1` uses an ASpeed BMC VGA controller (`46:00.0`) for host console video. This means blacklisting `amdgpu` does **not** remove the host's ability to display video or log in locally. Always verify the host has an alternative console before blacklisting a GPU driver.
Always verify the host has an alternative console (e.g. an ASpeed BMC VGA controller, a separate low-end GPU, or serial console) before blacklisting the primary GPU driver. Blacklisting `amdgpu` must not remove the host's ability to display video or log in locally.
### 6.7 IOMMU groups are clean, no ACS override needed
The GPU is in IOMMU group `30` and the audio function is in group `31`. There are no other devices in either group. This is the ideal scenario for passthrough — no ACS override patch or risky kernel parameters are needed.
For this setup, the GPU was the only device in its IOMMU group and the audio function was the only device in its group. This is the ideal scenario for passthrough — no ACS override patch or risky kernel parameters are needed. If your system groups the GPU with other devices, additional steps may be required.
### 6.8 Do not create the VM as part of this SOP
@ -305,7 +305,7 @@ This SOP intentionally stops at host preparation. VM creation, guest OS installa
---
## 7. Files changed on `mach1`
## 7. Files changed on the Proxmox node
| File | Purpose |
|------|---------|
@ -321,8 +321,8 @@ This SOP intentionally stops at host preparation. VM creation, guest OS installa
To restore the original host behavior and let the R9700 be used by the host again:
```bash
# Restore GRUB command line
sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="quiet amdgpu.ppfeaturemask=0xffffffff"/' /etc/default/grub
# Restore GRUB command line (replace with your original line as needed)
sed -i 's/^GRUB_CMDLINE_LINUX_DEFAULT=.*/GRUB_CMDLINE_LINUX_DEFAULT="quiet"/' /etc/default/grub
# Remove the new config files
rm /etc/modules-load.d/vfio.conf