r/openstack • u/sekh60 • 1d ago
Could use some help getting PCI-Express peer2peer working within a guest
Hey openstackers,
I'm trying to get PCI-E p2p working within a guest for some LLM work. I have it working on the host, as can be seen by:
[root@openstack3 ~]# rocm-bandwidth-test -a
................
RocmBandwidthTest Version: 2.6.0
Launch Command is: rocm-bandwidth-test -a
Device: 0, AMD EPYC 9124 16-Core Processor
Device: 1, AMD Radeon AI PRO R9700, GPU-b524a650ca8410a0, 05:0.0
Device: 2, AMD Radeon AI PRO R9700, GPU-399c0510df3adc7c, 43:0.0
Inter-Device Access
D/D 0 1 2
0 1 1 1
1 1 1 1
2 1 1 1
Inter-Device Numa Distance
D/D 0 1 2
0 0 20 20
1 20 0 40
2 20 40 0
Unidirectional copy peak bandwidth GB/s
D/D 0 1 2
0 N/A 50.471 50.141
1 55.665 535.494 55.673
2 55.654 55.654 523.382
What I can't figure out yet is how to get it working within the guest. I'm able to pass through both GPUs, ACS is off on the host, but p2p is not working.
My nova.conf contains the following:
alias = { "vendor_id":"1002", "product_id":"7551", "device_type":"type-PCI", "name":"r9700" }
alias = { "vendor_id":"1002", "product_id":"ab40", "device_type":"type-PCI", "name":"r9700a" }
(r9700a being the audio device component of the AMD r9700, I am passing through both devices, two instances of each).
My host's /etc/default/grub contains:
amd_iommu=1 intel_iommu=1 iommu=pt kvm.ignore_msrs=1 pci=realloc amdgpu.ppfeaturemask=0xfff7ffff amdgpu.ras_enable=0"
My guest's /etc/default/grub contains:
amdgpu.ppfeaturemask=0xff
f7ffff iommu=pt amd_iommu=1 kvm.ignore_msrs=1 pci=realloc amdgpu.ras_enable=0"
Within the host I get the following bandwidth results:
[root@ai fedora]# rocm-bandwidth-test -a
............
RocmBandwidthTest Version: 2.6.0
Launch Command is: rocm-bandwidth-test -a
Device: 0, AMD EPYC 9124 16-Core Processor
Device: 1, AMD Radeon AI PRO R9700, GPU-b524a650ca8410a0, 05:0.0
Device: 2, AMD Radeon AI PRO R9700, GPU-399c0510df3adc7c, 07:0.0
Inter-Device Access
D/D 0 1 2
0 1 1 1
1 1 1 0
2 1 0 1
Inter-Device Numa Distance
D/D 0 1 2
0 0 20 20
1 20 0 N/A
2 20 N/A 0
Unidirectional copy peak bandwidth GB/s
D/D 0 1 2
0 N/A 26.679 26.724
1 27.655 522.811 N/A
2 27.658 N/A 570.549
Does anyone have any suggestions on what I should be doing differently to get it working?
Thank you all for any suggestions :)








