Introduction: The Shift to Asymmetric NVM
Emerging Non-Volatile Memory (NVM) technologies, such as Intel’s 3D X-Point, have long promised a revolution in data center architecture by offering the persistence of storage with the byte-addressable speed of DRAM. Traditionally, NVM has been deployed in a symmetric architecture, where it is directly attached to a server's memory bus. While this provides the lowest possible local latency, it introduces significant operational challenges in large-scale data centers, including low hardware utilization, high availability risks, and severe memory pressure on local DRAM.
To address these issues, this paper proposes AsymNVM, a framework that transitions NVM from a locally attached component to a disaggregated, shared resource. By decoupling compute from storage, AsymNVM allows multiple "front-end" servers to access a centralized "back-end" NVM pool over a high-speed network using Remote Direct Memory Access (RDMA).
The Problem with Symmetric NVM
In a symmetric setup, NVM is often underutilized because a single server rarely requires the massive capacity (terabytes) that NVM modules provide. Furthermore, if a server fails, the persistent data stored on its local NVM becomes inaccessible, even if the memory hardware itself is perfectly functional. To mitigate this, systems often resort to replication, which requires keeping a full copy of the data structure in local volatile DRAM to maintain performance. This "double-buffering" effectively halves the available memory and limits the size of the working set a single node can manage.
The AsymNVM Framework
AsymNVM addresses these bottlenecks by treating NVM as a standalone "blade." The framework is built on several key technical innovations designed to minimize the overhead of network-based memory access:
1. The Gather-Apply Model
Rather than treating remote NVM as a simple extension of local memory, AsymNVM uses a two-phase access model. In the Gather phase, the front-end fetches necessary data, utilizing a local DRAM cache to avoid network round-trips for "hot" data (like the upper levels of a B+tree). In the Apply phase, modifications are pushed back to the NVM nodes.
As shown in the workflow above, the system intelligently manages data movement:
- Read (1): The front-end first checks the local memory cache.
- Swap In (2): If the data is missing, it is fetched from the back-end NVM.
- Direct Read (3): For specific operations, the front-end can read directly from NVM via RDMA.
- Append Log (5): Writes are first recorded as logs on the back-end to ensure persistence before being applied to the main data structure (6).
2. Operation Logging vs. Memory Logging
A critical optimization in AsymNVM is the use of operation logs (e.g., "Insert Key X") instead of traditional memory logs (which record raw address changes). Operation logs allow the front-end to batch multiple logical updates into a single RDMA write, significantly reducing the number of network messages required for complex data structure updates.
3. Two-Tier Memory Allocation
To avoid the latency of synchronous network calls for every memory allocation, AsymNVM employs a two-tier slab allocator. The back-end manages large "slabs" of NVM, while the front-end manages finer-grained allocations within those slabs locally. This allows for near-instantaneous memory allocation for most operations.
Evaluation and Key Results
The authors evaluated AsymNVM using a suite of eight common data structures and industry-standard benchmarks like TATP and SmallBank. The results were compelling:
- Performance Gains: AsymNVM achieved a 6× to 22× speedup compared to a naive RDMA implementation that simply replaced local loads/stores with remote ones.
- Symmetric Parity: Remarkably, the framework achieved performance comparable to local symmetric NVM architectures. This suggests that the overhead of the network can be almost entirely masked by efficient caching and operation logging.
- Scalability: The system demonstrated near-linear scalability for read-heavy workloads, proving that a disaggregated NVM pool can effectively serve a large cluster of compute nodes without becoming a bottleneck.
Conclusion
AsymNVM demonstrates that disaggregating NVM is not only feasible but highly efficient. By shifting to an asymmetric model and utilizing a passive back-end, data centers can achieve higher resource utilization and better availability without sacrificing the performance benefits of persistent memory. While challenges remain regarding network congestion and recovery complexity for massive datasets, AsymNVM provides a robust blueprint for the next generation of disaggregated data center storage.
Summary updated