Introduction
As machine learning models scale to tens of thousands of GPUs, the network infrastructure—the switches, NICs, and cables connecting them—has become a massive bottleneck. Testing this infrastructure is notoriously difficult due to the "GPU Tax" and the "Simulation Gap." Traditional simulators often fail to model proprietary hardware accurately and cannot predict real-world "gremlins" like a degrading NIC or a misconfigured buffer.
The GENIE Framework
GENIE (GPUs Eliminated for Network Infrastructure Examination) is a testing framework that allows researchers to evaluate large-scale ML network performance on real hardware without needing a single expensive GPU. The core innovation is the decoupling of computation from communication.
The framework consists of two main parts:
- The Workload Feeder (Modified ASTRA-sim): Instead of running heavy math on a GPU, GENIE uses the ASTRA-sim simulator to manage the logic of the ML job by reading a Chakra graph. For compute tasks, it simply "sleeps" for the duration a real GPU would take.
- The Traffic Generator: When it hits a communication task, it triggers the host CPU to inject actual RDMA traffic into the network. This traffic traverses real NICs and switches, mimicking the size and timing of real ML traffic.
Key Insights and Results
GENIE provides realism over pure simulation by capturing hardware anomalies that simulators miss. Because it is dependency-aware, if one part of the network slows down, the "simulated" training job slows down in response, capturing the ripple effect of network delays on total training time.
Initial results show that GENIE can:
- Capture the impact of hardware events like packet drops and backpressure.
- Replicate the behavior of distributed training ranks across multiple nodes.
- Be ported to various high-performance network fabrics (Infiniband, RoCE, Slingshot) without rewriting ML model code.
Limitations and Future Work
Future work focuses on validating how well a CPU can mimic the specific burstiness and PCIe-interaction patterns of a GPU. Currently, GENIE focuses on inter-node network traffic and does not yet model intra-node interconnects like NVLink. The team plans to build out the full implementation and run it against large-scale GPU "ground truth" data to prove its accuracy across different failure scenarios.
Summary updated