PAM: When Overloaded, Push Your Neighbor Aside!

Zili Meng, Jun Bi, Chen Sun, Shuhe Wang, Minhu Wang, Hongxin Hu

Introduction

In modern data centers, Network Function Virtualization (NFV) has replaced traditional hardware middleboxes with software-based Virtual Network Functions (vNFs). To handle high-speed traffic, these vNFs are often offloaded from general-purpose CPUs to specialized Smart Network Interface Cards (SmartNICs). However, unpredictable traffic patterns can lead to "hot spots" where a specific vNF on a SmartNIC becomes overloaded.

The Challenge: The PCIe Bottleneck

The intuitive solution to an overloaded vNF is to migrate that specific bottlenecked function back to the CPU. However, service chains require packets to traverse vNFs in a strict sequence. Moving a middle-of-the-chain vNF to the CPU forces packets to cross the PCIe bus multiple times (SmartNIC → CPU → SmartNIC), as shown in Figure 3(b). Since each PCIe crossing adds significant latency, this "naive" approach often degrades performance for latency-sensitive applications.

Comparison of service chain migration strategies: (a) original state, (b) naive migration causing extra PCIe crossings, and (c) PAM's neighbor-pushing approach.

The PAM Solution: Push Aside Migration

The researchers propose PAM (Push Aside Migration), a strategy that shifts focus from the bottlenecked vNF to its neighbors. Instead of moving the overloaded function, PAM identifies "border" vNFs—those already at the transition point between the SmartNIC and the CPU.

The PAM algorithm follows three key steps:

  1. Border vNF Identification: It targets vNFs at the hardware boundary. Migrating these does not introduce new PCIe crossings; it merely shifts the existing boundary.
  2. Lightweight Migration Selection: PAM selects the border vNF with the lowest capacity requirement to migrate first, ensuring minimal overhead.
  3. Overload Alleviation Check: The system mathematically verifies that the migration provides sufficient relief to the SmartNIC without overloading the CPU.

By "pushing" a neighbor aside, the bottlenecked vNF gains the necessary resources on the SmartNIC to maintain high performance.

Experimental Evaluation

The system was evaluated using a hardware testbed featuring:

  • Hardware: A server with a Netronome Agilio CX 2×10GbE SmartNIC and dual Intel Xeon E5-2620 v2 CPUs.
  • Traffic: DPDK-based generation with packet sizes from 64B to 1500B.
  • Service Chain: A sequence consisting of a Firewall, Logger, Monitor, and Load Balancer.

Results

The evaluation compared PAM against the naive migration strategy.

Performance comparison of the Original, Naive, and PAM strategies in terms of (a) Latency and (b) Throughput.
  • Latency: PAM achieved an 18% average reduction in latency compared to the naive solution. Crucially, PAM's latency remained nearly identical to the pre-overload state because it avoided redundant PCIe crossings.
  • Throughput: PAM showed a slight improvement in throughput, effectively rebalancing the workload across the CPU and SmartNIC architectures.

Conclusion and Future Work

PAM demonstrates that in high-performance networking, the physical bus (PCIe) is often a more critical constraint than raw compute power. By maintaining holistic awareness of the service chain, PAM preserves low latency during traffic spikes. Future research will focus on extending PAM to FPGA-based SmartNICs and refining performance models for more complex, branching service chains.

Summary updated

Abstract

Recently SmartNICs are widely used to accelerate service chains in NFV. However, when the SmartNIC is overloaded, casually migrating vNFs away from SmartNIC to CPU may lead to additional packet transmissions between SmartNIC and CPU. To address this problem, we present PAM, push aside migration to effectively alleviate the hot spot on SmartNIC with no performance overhead. Our key novelty is to push vNFs on the border of SmartNIC and CPU aside to release resources for the bottleneck vNF. Evaluation shows that PAM could efficiently alleviate the hot spot on SmartNIC and generate a service chain with much lower latency compared with the naive solution.
BibTeX
@inproceedings{meng2018pam,
  author = {Zili Meng and Jun Bi and Chen Sun and Shuhe Wang and Minhu Wang and Hongxin Hu},
  title = {{PAM: When Overloaded, Push Your Neighbor Aside!}},
  booktitle = {SIGCOMM Posters and Demos},
  year = {2018},
  doi = {10.1145/3234200.3234215},
}
Figure