This is my post. Basically, I needed P2P between four R9700s for tensor parallel serving, which means a 32 GB BAR on each card. With all six GPUs in, the BIOS couldn't map them, and Linux sized the switch's window too small, so only two cards got their BARs. The module in the post works around that. With P2P off, prefill was about half as fast.
The bottleneck with AI and hardware is really pushing things.
Even with six capable GPUs that have plenty of VRAM with enough PCIe bandwidth, you then discover that memory address allocation is preventing the hardware from doing what it's physically capable of.
Most CPUs can address 48 bits of physical memory (hundreds of terabytes), why is that even a problem ? Is it just some hardware limitation somewhere else or badly written firmware ?
Yeah, it's the firmware. It's an AM4 board, and the BIOS only gives PCI devices the space from the top of RAM up to 512 GB. Each 32 GB BAR has to start on a 32 GB boundary, and its small doorbell BAR spills into the next slot, so each card needs 64 GB. Six won't fit. So I turn Resizable BAR off in the BIOS and resize the four R9700s from Linux, where I hit the kernel bug.
It feels ridiculous that a person has to go to these kind of lengths to do what is in the end a trivial thing: assigning little pieces of a gigantic virtual address space to specific peripherals.
The only thing that's complex here is having to influence a bunch of hidden software through three layers of indirection. The software is all standing in the way of a person just writing some values to registers.
Another important benefit is that transfers between cards flow across the switch and bypass the CPU, that ain't nothing. At least for the AMD cards. Nvidia firmware refuses nach
Asking because I really want to see your kit.
I think a lot of folks use open air aluminum extrusion frames that were once meant for mining.
Even with six capable GPUs that have plenty of VRAM with enough PCIe bandwidth, you then discover that memory address allocation is preventing the hardware from doing what it's physically capable of.
I wonder if AI could RE the BIOS enough to fix that limit
The only thing that's complex here is having to influence a bunch of hidden software through three layers of indirection. The software is all standing in the way of a person just writing some values to registers.