Performance Analysis of Packet Receiving Process in Linux Kernel

Xiaobo Ji, Zeng Fan, Min ZhangJournal of Convergence Information Technology 2012 · DOI

TL;DR

This paper analyzes how Linux kernels handle incoming network traffic, particularly focusing on how to better utilize multi-core processors. It introduces and evaluates Receive Packet Steering (RPS) and Receive Flow Steering (RFS), two features in Linux kernel 2.6.35 that distribute network load across multiple CPUs, significantly improving throughput and reducing packet loss compared to older kernel versions.

What Problem Does This Paper Address?

Modern network interface cards (NICs) can process an enormous number of packets, overwhelming a single CPU. While CPU core counts have increased, individual CPU speeds have plateaued. Traditional Linux network stacks struggle to efficiently distribute this incoming network load across multiple cores, leading to performance bottlenecks and degraded throughput. This paper aims to analyze the packet receiving process in Linux and evaluate new features designed to address this challenge by enabling better multi-CPU utilization.

Methods

The paper first details the packet receiving process in older Linux kernels (2.6.32), distinguishing between non-NAPI and NAPI (New API) modes. NAPI is a more efficient interrupt-handling mechanism that allows the kernel to poll for packets rather than relying solely on interrupts, reducing overhead.

The core of the paper then focuses on two new features introduced in Linux kernel 2.6.35:

  • Receive Packet Steering (RPS): This technique uses a hash function on packet headers to determine which CPU should process a given packet. The goal is to distribute packets across CPUs, leveraging cache locality by aiming to send packets from the same flow to the same CPU. RPS can be implemented without requiring changes to device drivers.
  • Receive Flow Steering (RFS): This is an extension of RPS that aims to prevent out-of-order packet delivery, a potential issue with RPS. RFS uses two hash tables: one to determine the "desired" CPU based on the application's current CPU, and another to track the "current" CPU that last handled a flow. It prioritizes sending packets to the desired CPU if certain conditions are met, ensuring packets for a flow are processed in order.

The authors then conducted experiments using the netperf tool to compare the performance of Linux kernel 2.6.35 with and without RPS/RFS enabled.

Datasets

The paper does not explicitly mention specific datasets. Instead, it uses a benchmark tool called netperf to evaluate performance metrics. The experiments involved running netperf tests (specifically TCP_RR with 1-byte requests and responses) on customized Linux kernel 2.6.35 builds, both with and without RPS/RFS enabled.

Key Insights and Ideas

  • The Bottleneck is CPU Distribution: The primary limitation in high-speed networking isn't just packet processing speed, but the ability to distribute that processing across available CPU cores.
  • NAPI Improves Efficiency: NAPI is a significant improvement over traditional interrupt-driven packet handling, allowing for more efficient processing by polling.
  • RPS Distributes Load: RPS effectively spreads the incoming network traffic across multiple CPUs by using packet header hashes, leading to higher throughput.
  • RFS Solves Out-of-Order Issues: RFS builds upon RPS to ensure packets within the same flow are processed in order, preventing a common drawback of simpler load-balancing schemes.
  • Driver-Independent Benefits: A key advantage of RPS is its ability to work without requiring modifications to existing network device drivers.

Key Results

The experimental results demonstrate a significant performance improvement with RPS and RFS enabled:

  • Throughput: Linux kernel 2.6.35 configured with RPS/RFS showed substantially higher throughput compared to the same kernel without these features. For example, one benchmark showed a jump from 90K transactions per second to 285K transactions per second with RPS enabled on an 8-core Intel system with a tg3 NIC.
  • Transaction Rate: Similar improvements were observed in transaction rates.
  • Drop Packet Rate: RPS/RFS significantly reduced the packet drop rate in high-speed network scenarios.
  • Average Latency: Linux kernel 2.6.35 with RPS/RFS greatly outperformed the version without these features in terms of average latency.

Limitations and Future Work

  • Driver-Specific Hash Calculation: While RPS can function without driver changes, the paper notes that NICs capable of calculating a hash value themselves can further improve performance by eliminating extra cache misses.
  • Out-of-Order Packets with RPS: The paper acknowledges that RPS alone can lead to out-of-order packets, which is why RFS was developed to address this.
  • Future Work: The paper's conclusion implies that further optimization and analysis of these multi-CPU packet handling techniques would be beneficial. The focus on physical and data link layers suggests that deeper analysis into higher network layers could also yield insights.

Summary updated

Abstract

Because of improved high-bandwidth, contemporary network cards can move a lot of packets to the point that it is hard for a single modern CPU to keep up. Moreover, in recent years, CPU speed almost has stopped improving, but the number of CPU cores is increasing now. In this case, the Linux kernel must be capable of distributing the work across multiple CPUs in order to improve throughput for high-speed network systems. Two new features contributed by Google have been introduced into the Linux kernel 2.6.35 in order to spread the load of network handling across multiple CPUs, which are receive packet steering and receive flow steering, respectively. In this paper, the packet receiving process in Linux kernel is analyzed only including physical layer and data link layer. We also study the receive packet steering and receive flow steering introduced in Linux kernel 2.6.35. Finally, we performed various experiments and proved the superiority of receive packet steering and receive flow steering.
BibTeX
@article{ji2012performance,
  author = {Xiaobo Ji and Zeng Fan and Min Zhang},
  title = {{Performance Analysis of Packet Receiving Process in Linux Kernel}},
  journal = {Journal of Convergence Information Technology},
  year = {2012},
  doi = {10.4156/jcit.vol7.issue17.31},
}
Figure