Enabling Kernel Bypass Networking on gem5
Abstract
Full-system simulation of computer systems is critical to capture the complex interplay between various hardware and software components in future systems. Modeling the network subsystem is indispensable to the fidelity of the full-system simulation due to the increasing importance of scale-out systems. The network software stack has undergone major changes over the last decade, and kernel-bypass networking stacks and data-plane networks are rapidly replacing the conventional kernel network stack. Nevertheless, the current state-of-the-art architectural simulators still use kernel networking which precludes realistic network application scenarios. In this work, we enable kernel bypass networking stack on gem5, the state-of-the-art full-system architectural simulator. We extend gem5's NIC hardware model and device driver to enable the support for userspace device drivers to run the DPDK framework. We also implement a network load generator hardware model in gem5 to generate various traffic patterns and perform per-packet timestamp and latency measurements without introducing packet loss. Our experimental results show that DPDK's simulated network bandwidth scales with the number of cores and NIC ports. As two use cases, we analyze the sensitivity of (1) network performance to several microarchitectural parameters, and (2) direct cache access (DCA) technology to DPDK burst size.
Introduction
In modern datacenters, network speeds are rapidly approaching terabit-per-second rates. To maintain efficiency at these speeds, software has transitioned toward "kernel bypass" techniques, such as the Data Plane Development Kit (DPDK). These techniques allow applications to interact directly with network hardware, circumventing the overheads associated with the operating system kernel.
However, architectural simulators like gem5 have traditionally relied on the standard Linux network stack. This stack is characterized by frequent system calls, high interrupt latency, and excessive data copying, which limits simulated network performance to approximately 10-20 Gbps. This software-induced bottleneck masks the true performance of the underlying hardware, preventing researchers from accurately studying how future processors will behave in high-speed, 100+ Gbps environments.
Architectural Enhancements
To address this, the researchers modified gem5 to support high-performance kernel bypass networking. Key enhancements included updating the PCI model to support "interrupt disable" bits and byte-granular access, which are essential for userspace drivers. The emulated Intel 8254x NIC was improved with an Interrupt Mask Register and a user-controllable threshold for descriptor writebacks to manage memory pressure during high-speed DMA.
Furthermore, a new hardware traffic generator, EtherLoadGen, was introduced. This model injects packets at precise rates and measures per-packet latency without the computational cost of simulating an entire second node, significantly accelerating experimental cycles.
Performance and Scalability
The implementation of DPDK within gem5 yielded a dramatic increase in simulated bandwidth. A single NIC port running DPDK sustained approximately 53 Gbps, a 5.4x improvement over the ~10 Gbps achieved with the standard Linux iperf setup.
As shown in Figure 4(a), DPDK performance scales effectively with hardware resources. Adding a fourth NIC port increased DPDK bandwidth by 24.1%, whereas the Linux kernel stack saw only a 5.3% gain due to internal overheads. Each simulated ARM core (Cortex-A72) was able to sustain roughly 25 Gbps of traffic without packet loss using the DPDK L2fwd application.
Microarchitectural Insights
A critical finding of this work is that the choice of software stack fundamentally changes architectural conclusions. As illustrated in Figure 4(b), increasing CPU frequency significantly improved the performance of the computationally expensive kernel stack (32.5% gain) but had a negligible effect on DPDK (1.2% gain). This demonstrates that using an outdated software stack in a simulator can lead to incorrect assumptions about hardware optimizations.
By enabling kernel bypass, the simulation shifts the performance bottleneck from the software stack to the processor and memory hierarchy. This allows for nuanced studies, such as analyzing how large DPDK batch sizes cause contention in the Last Level Cache (LLC) when using Direct Cache Access (DCA).
Conclusion
This framework enables gem5 to accurately model modern, high-speed network scenarios by removing the software bottleneck of the standard kernel stack. Future work includes expanding support for a wider variety of NIC models and continuing the effort to upstream these enhancements into the main gem5 repository.
The source code is available at: https://github.com/agsiddharth/CAL-DPDK-GEM5
Summary updated