Abstract
This paper re-evaluates the performance implications of Ethernet "Jumbo Frames" (MTUs up to 9000 bytes) in modern data center environments. While historical networking wisdom emphasized the necessity of jumbo frames for CPU efficiency, the advent of hardware offloading features like Large Send Offload (LSO) and Large Receive Offload (LRO) has narrowed the performance gap. Through an empirical study of bulk transfers, MapReduce workloads, and tiered web services, we demonstrate that jumbo frames still provide measurable benefits in throughput and CPU utilization. Furthermore, we identify and resolve a critical latency issue involving the interaction between large MTUs and Nagle’s algorithm.
Introduction
For over a decade, the use of Jumbo Frames has been a staple recommendation for high-performance networking. By increasing the Maximum Transmission Unit (MTU) from the standard 1500 bytes to 9000 bytes, the overhead associated with packet processing is theoretically reduced. However, modern Network Interface Cards (NICs) have evolved to include sophisticated offloading capabilities. Features such as LSO and LRO allow hardware to aggregate packets, potentially rendering the manual configuration of jumbo frames redundant or even counterproductive due to increased serialization delay. This study seeks to determine if the operational complexity of maintaining jumbo frames is still justified in the modern era.
Experimental Methodology
The researchers utilized a dedicated testbed consisting of 12 servers interconnected via 1Gbps links. The evaluation compared four MTU sizes: 1500 bytes, 4KB, 8KB, and 9KB. To capture a realistic picture of data center traffic, the study moved beyond micro-benchmarks to evaluate three distinct application architectures.
The workloads included bulk data transfers (FTP), data-intensive computing (Hadoop MapReduce running Terasort and Grep), and latency-sensitive tiered web services (Olio toolkit). System-level performance was monitored using the Linux perf tool to track CPU cycles, instructions per cycle (IPC), and interrupt frequency.
Performance Analysis
The results indicate that while hardware offloading mitigates some of the traditional bottlenecks, jumbo frames still offer significant advantages.
Throughput and CPU Efficiency
For 1MB file transfers, jumbo frames achieved a 10% throughput improvement over standard frames. This benefit scaled to a 5.5% improvement even at full link saturation with 100MB files. More importantly, jumbo frames reduced CPU cycle consumption by approximately 13% and decreased network interrupts by 6%, freeing up valuable compute resources for application logic. In Hadoop environments, the use of 9KB frames reduced the completion time of a 100GB Terasort by nearly 40 seconds.
The Nagle Interaction and Latency
A significant finding was the initial observation of increased response times in web applications when using jumbo frames. Investigation revealed an adverse interaction with Nagle’s algorithm. Because Nagle’s algorithm waits for a buffer to reach the Maximum Segment Size (MSS) before transmission, the larger MSS of jumbo frames caused longer buffering delays. Once Nagle’s algorithm was disabled, response times for the Olio web service became identical across all MTU sizes, confirming that jumbo frames do not inherently degrade latency.
Conclusion
Jumbo frames remain a valuable tool for optimizing "East-West" data center traffic. While the "miraculous" gains of the past have been tempered by modern NIC offloading, the 13% reduction in CPU overhead and improved throughput under congestion justify their use. Operators can safely enable jumbo frames for intra-data center traffic, provided that latency-sensitive applications are configured to disable Nagle’s algorithm to avoid buffering delays. Future research should extend these findings to 10Gbps and 40Gbps environments where CPU bottlenecks are likely more pronounced.
Summary updated