RFP: A Remote Fetching Paradigm for RDMA-Accelerated Systems

Mingxing Zhang (章明星), Maomeng Su, Mingxing Zhang, Kang Chen, Yongwei Wu, Guoliang LiarXiv 2015 · arXiv

Introduction

In modern datacenter environments, high-performance applications like in-memory key-value stores are increasingly bottlenecked by Input/Output Operations Per Second (IOPS) rather than raw bandwidth. While Remote Direct Memory Access (RDMA) has emerged as a solution to bypass CPU overhead, existing design paradigms often fail to fully exploit the underlying hardware.

Traditional RDMA implementations typically follow one of two models: the Server-Bypass Model, where clients directly access server memory (raising security and consistency concerns), or the Server-Reply Model, where the server processes requests and "pushes" results back via RDMA-writes. This research identifies a fundamental hardware asymmetry in Network Interface Cards (NICs): serving an incoming RDMA-read is significantly more efficient than issuing an outgoing RDMA-write. On tested hardware, in-bound reads were found to be up to five times faster than out-bound writes. To exploit this, the authors propose the Remote Fetching Paradigm (RFP).

Methodology: The Remote Fetching Paradigm

RFP flips the traditional data flow to leverage NIC hardware logic. Instead of the server pushing data, the client proactively "pulls" it. The process follows five distinct steps:

  1. Request: The client sends a request to the server using an RDMA-write.
  2. Buffering: The server NIC places the request into a memory buffer.
  3. Processing: The server CPU processes the request and writes the result into a client-specific response buffer.
  4. Staging: The result is held in the response buffer.
  5. Fetch: The client performs an in-bound RDMA-read on the server’s NIC to retrieve the result.
The RFP architecture showing the 5-step request and fetch process.

To demonstrate RFP, the authors developed Jakiro, an in-memory key-value store. Jakiro optimizes the fetch phase by using a "combined status checking and result fetching" technique. Clients perform a single RDMA-read of a fixed size (the Remote Fetching Size) to retrieve both the status header and the data simultaneously, minimizing round-trip latency.

Evaluation and Key Results

The researchers evaluated Jakiro using the Yahoo! Cloud Serving Benchmark (YCSB) with 128 million items and a real-world Weibo dataset consisting of 50 million posts. The evaluation focused on throughput (IOPS) and latency across varying thread counts and data distributions.

Jakiro demonstrated significant performance gains over traditional models. It reached a peak throughput of 5.5 million operations per second (MOPS) for 32-byte items, representing a 158% improvement over the standard Server-Reply model and a 310% increase over RDMA-Memcached.

Throughput of Jakiro in MOPS as a function of the number of client threads.

In addition to throughput, Jakiro achieved an average latency of 5.78µs, outperforming the Server-Reply model by 108%. The system maintained these advantages even under highly skewed (Zipfian) workloads, as the offloading of networking logic to the client-side fetching mechanism prevented the server NIC from becoming a bottleneck.

Conclusion

The Remote Fetching Paradigm successfully addresses the hardware limitations of the Server-Reply model by shifting the responsibility of data transfer to the clients. By leveraging the efficiency of in-bound RDMA-reads, RFP provides a high-performance, secure alternative to server-bypass models. While the advantage is most pronounced for small payloads (< 1.5 KB), RFP represents a significant step forward in optimizing RDMA-accelerated systems for the IOPS-intensive demands of modern cloud workloads.

Summary updated

BibTeX
@misc{2015rfp,
  author = {Mingxing Zhang (章明星) and Maomeng Su and Mingxing Zhang and Kang Chen and Yongwei Wu and Guoliang Li},
  title = {{RFP: A Remote Fetching Paradigm for RDMA-Accelerated Systems}},
  year = {2015},
  eprint = {1512.07805},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1512.07805},
}
Figure