Thin servers with smart pipes

Kevin Lim, David Meisner, Али Саиди, Parthasarathy Ranganathan, Thomas F. WenischACM SIGARCH Computer Architecture News 2013 · DOI

TL;DR

This paper argues that current server architectures are inefficient for memcached, a critical component of modern internet services. They propose "Thin Servers with Smart Pipes" (TSSP), an SoC design that offloads GET requests to a dedicated hardware accelerator, achieving a 6-16x improvement in power-performance compared to conventional servers.

What Problem Does This Paper Address?

Modern internet services rely heavily on distributed in-memory key-value stores like memcached for scalability. However, current deployments use expensive, high-end processors that are surprisingly inefficient for memcached workloads. This inefficiency stems not from raw compute or memory bandwidth limitations, but from the overhead of processing network packets within the OS kernel and NIC. This leads to wasted power and limits the overall capacity of data centers, which are often constrained by power delivery. The paper aims to identify these bottlenecks and propose a more efficient architecture.

Methods

The authors conducted a detailed architectural characterization of memcached performance and power consumption. They used a custom load-testing framework to simulate realistic traffic patterns, varying object sizes and access frequencies. They then analyzed performance counters and microarchitectural metrics on both high-performance (Xeon) and low-power (Atom) server configurations with different Network Interface Cards (NICs). Based on these findings, they propose the Thin Servers with Smart Pipes (TSSP) architecture, which integrates a low-power CPU with a dedicated hardware accelerator for memcached GET requests within a System-on-Chip (SoC). This accelerator handles network packet processing and data lookups, offloading these tasks from the CPU. They evaluated the TSSP concept using an FPGA prototype.

Datasets

The paper does not explicitly list or use pre-existing datasets for training. Instead, they created a suite of five carefully designed memcached workloads to emulate real-world traffic. These workloads vary in object size distribution (e.g., FixedSize, MicroBlog, Wiki, ImgPreview, FriendFeed) and access patterns (e.g., single GET vs. MULTI-GET requests), aiming to expose a broad range of memcached behavior and performance characteristics. They state that these workloads exhibit similar statistical properties to those reported for production clusters at Facebook.

Key Insights and Ideas

  • Network Stack is the Bottleneck: The primary performance limitation for memcached is not CPU processing power or memory bandwidth, but the overhead of the OS kernel and NIC in processing each network packet.
  • Inefficient CPU Utilization: Even high-end CPUs are significantly underutilized, operating at a fraction of their theoretical instruction throughput due to frequent trips into the networking stack, leading to poor instruction cache and branch predictor performance.
  • Low-Power Cores Aren't a Silver Bullet: While low-power cores offer energy savings, their current implementations are too inefficient at handling the network-intensive memcached workload to provide a satisfactory power-performance trade-off.
  • NIC Quality Matters More Than Bandwidth: Advanced NIC features like segment offload and multi-queue support significantly improve performance, often more than simply increasing raw bandwidth.
  • Hardware Acceleration for GET Requests: Offloading the most frequent operation (GET requests) to a dedicated hardware accelerator within an SoC can drastically improve both performance and power efficiency.

Key Results

  • Significant Inefficiency: Xeon processors operate at less than 1/8th and Atom processors at less than 1/16th of their theoretical instruction throughput for memcached.
  • Instruction Cache Thrashing: Memcached exhibits an instruction cache miss rate up to 15x worse than typical benchmarks, primarily due to the large code footprint of the Linux kernel and networking stack.
  • NIC Feature Impact: Enterprise-class NICs with features like software batching and multi-queue support provide 70-110% performance improvements over commodity NICs.
  • TSSP Performance: The TSSP architecture, demonstrated on an FPGA prototype, achieves a 6-16x improvement in power-performance compared to conventional Xeon and Atom-based server baselines. For the FixedSize workload, TSSP achieved 282 KOps/s at 16W, while Xeon and Atom systems consumed 143W and 35W respectively, with lower throughput.
  • UDP vs. TCP: While using UDP instead of TCP can improve performance for some workloads, the primary gains in TSSP come from hardware acceleration, not just protocol choice.

Limitations and Future Work

  • FPGA Prototype Limitations: The TSSP evaluation was based on an FPGA prototype, which required some compromises, such as limiting key sizes and value sizes, and implementing memory management in software. The authors note that a custom ASIC implementation would likely yield further efficiency gains.
  • Software-Hardware Split: While TSSP offloads GET requests to hardware, other operations like SET, memory allocation, and eviction are handled by software. The authors acknowledge that implementing these complex operations in hardware would be challenging.
  • Future Work: The authors suggest that their TSSP design could be applied to other data- and network-centric workloads. They also imply that further optimization of the hardware accelerator and tighter integration within the SoC could lead to even greater improvements. The paper also hints at the potential for custom ASICs for maximum efficiency.

Summary updated

Abstract

Distributed in-memory key-value stores, such as memcached, are central to the scalability of modern internet services. Current deployments use commodity servers with high-end processors. However, given the cost-sensitivity of internet services and the recent proliferation of volume low-power System-on-Chip (SoC) designs, we see an opportunity for alternative architectures. We undertake a detailed characterization of memcached to reveal performance and power inefficiencies. Our study considers both high-performance and low-power CPUs and NICs across a variety of carefully-designed benchmarks that exercise the range of memcached behavior. We discover that, regardless of CPU microarchitecture, memcached execution is remarkably inefficient, saturating neither network links nor available memory bandwidth. Instead, we find performance is typically limited by the per-packet processing overheads in the NIC and OS kernel---long code paths limit CPU performance due to poor branch predictability and instruction fetch bottlenecks. Our insights suggest that neither high-performance nor low-power cores provide a satisfactory power-performance trade-off, and point to a need for tighter integration of the network interface. Hence, we argue for an alternate architecture---Thin Servers with Smart Pipes (TSSP)---for cost-effective high-performance memcached deployment. TSSP couples an embedded-class low-power core to a memcached accelerator that can process GET requests entirely in hardware, offloading both network handling and data look up. We demonstrate the potential benefits of our TSSP architecture through an FPGA prototyping platform, and show the potential for a 6X-16X power-performance improvement over conventional server baselines.
BibTeX
@inproceedings{lim2013thin,
  author = {Kevin Lim and David Meisner and Али Саиди and Parthasarathy Ranganathan and Thomas F. Wenisch},
  title = {{Thin servers with smart pipes}},
  booktitle = {ACM SIGARCH Computer Architecture News},
  year = {2013},
  doi = {10.1145/2508148.2485926},
}
Figure