Jumbo Frames or Not: That is the Question!

Pawan Prakash, Myungjin Lee, Yuchong Hu, Ramana Rao KompellaJTCVS 2013

Abstract

This paper re-evaluates the performance implications of Ethernet "Jumbo Frames" (MTUs up to 9000 bytes) in modern data center environments. While historical networking wisdom emphasized the necessity of jumbo frames for CPU efficiency, the advent of hardware offloading features like Large Send Offload (LSO) and Large Receive Offload (LRO) has narrowed the performance gap. Through an empirical study of bulk transfers, MapReduce workloads, and tiered web services, we demonstrate that jumbo frames still provide measurable benefits in throughput and CPU utilization. Furthermore, we identify and resolve a critical latency issue involving the interaction between large MTUs and Nagle’s algorithm.

Introduction

For over a decade, the use of Jumbo Frames has been a staple recommendation for high-performance networking. By increasing the Maximum Transmission Unit (MTU) from the standard 1500 bytes to 9000 bytes, the overhead associated with packet processing is theoretically reduced. However, modern Network Interface Cards (NICs) have evolved to include sophisticated offloading capabilities. Features such as LSO and LRO allow hardware to aggregate packets, potentially rendering the manual configuration of jumbo frames redundant or even counterproductive due to increased serialization delay. This study seeks to determine if the operational complexity of maintaining jumbo frames is still justified in the modern era.

Experimental Methodology

The researchers utilized a dedicated testbed consisting of 12 servers interconnected via 1Gbps links. The evaluation compared four MTU sizes: 1500 bytes, 4KB, 8KB, and 9KB. To capture a realistic picture of data center traffic, the study moved beyond micro-benchmarks to evaluate three distinct application architectures.

Figure 1: Application workloads used in the study: (a) File transfer types, (b) 3-tier web service, and (c) MapReduce.

The workloads included bulk data transfers (FTP), data-intensive computing (Hadoop MapReduce running Terasort and Grep), and latency-sensitive tiered web services (Olio toolkit). System-level performance was monitored using the Linux perf tool to track CPU cycles, instructions per cycle (IPC), and interrupt frequency.

Performance Analysis

The results indicate that while hardware offloading mitigates some of the traditional bottlenecks, jumbo frames still offer significant advantages.

Figure 2: Performance metrics including instruction counts, IPC, and throughput under varying RTT inflation.

Throughput and CPU Efficiency

For 1MB file transfers, jumbo frames achieved a 10% throughput improvement over standard frames. This benefit scaled to a 5.5% improvement even at full link saturation with 100MB files. More importantly, jumbo frames reduced CPU cycle consumption by approximately 13% and decreased network interrupts by 6%, freeing up valuable compute resources for application logic. In Hadoop environments, the use of 9KB frames reduced the completion time of a 100GB Terasort by nearly 40 seconds.

The Nagle Interaction and Latency

A significant finding was the initial observation of increased response times in web applications when using jumbo frames. Investigation revealed an adverse interaction with Nagle’s algorithm. Because Nagle’s algorithm waits for a buffer to reach the Maximum Segment Size (MSS) before transmission, the larger MSS of jumbo frames caused longer buffering delays. Once Nagle’s algorithm was disabled, response times for the Olio web service became identical across all MTU sizes, confirming that jumbo frames do not inherently degrade latency.

Conclusion

Jumbo frames remain a valuable tool for optimizing "East-West" data center traffic. While the "miraculous" gains of the past have been tempered by modern NIC offloading, the 13% reduction in CPU overhead and improved throughput under congestion justify their use. Operators can safely enable jumbo frames for intra-data center traffic, provided that latency-sensitive applications are configured to disable Nagle’s algorithm to avoid buffering delays. Future research should extend these findings to 10Gbps and 40Gbps environments where CPU bottlenecks are likely more pronounced.

Summary updated

BibTeX
@inproceedings{prakash2013jumbo,
  author = {Pawan Prakash and Myungjin Lee and Yuchong Hu and Ramana Rao Kompella},
  title = {{Jumbo Frames or Not: That is the Question!}},
  booktitle = {JTCVS},
  year = {2013},
}
Figure