Disaggregation and the Application

Sebastian Angel, Mihir Nanavati, Siddhartha Sen

TL;DR

This paper argues that disaggregated data centers (DDCs) should expose hardware resource disaggregation to applications, rather than abstracting it away. They propose new OS interfaces for memory reassignment and failure notification, demonstrating how these can significantly improve performance and resilience for data-parallel and fault-tolerant applications.

What Problem Does This Paper Address?

Modern data centers are increasingly moving towards disaggregated architectures, where resources like compute, memory, and storage are pooled and accessed over a high-speed interconnect. This offers operational benefits like better resource utilization and flexibility. However, existing research in this area has largely focused on making these disaggregated resources transparent to applications, preserving traditional OS abstractions. The authors argue this approach misses a significant opportunity. They believe that by exposing the disaggregated nature of the hardware – specifically, the ability to dynamically reassign memory and the failure independence of components – applications can be designed to exploit these features for substantial performance and resilience gains. This paper tackles the challenge of designing OS interfaces that enable this application-driven exploitation of disaggregation.

Methods

The core of the paper's contribution lies in proposing and discussing three novel OS interfaces for disaggregated environments:

  • Memory Grant: This allows an application to voluntarily "gift" its memory pages to another application. The authors envision this as a mechanism for efficient data transfer, enabling zero-copy operations by remapping memory directly.
  • Memory Steal: This is an involuntary reassignment of memory from one application to another. While similar to a grant from the interconnect's perspective, the key difference is the lack of prior warning to the source. This is crucial for failure recovery, allowing a surviving replica to take over the memory of a failed one.
  • Failure Notification: Applications can opt-in to receive explicit notifications when memory components fail. This allows for faster and more proactive failure recovery than relying on traditional timeouts.

These interfaces are discussed within the context of a proposed DDC architecture featuring "blades" with compute and memory elements connected via a programmable "Rack MMU" (Memory Management Unit). The paper also explores how these primitives can be implemented and integrated into existing OSes.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Transparency is Not Always Best: The paper challenges the prevailing wisdom that disaggregation should be hidden from applications. They argue that exposing this reality can unlock significant benefits.
  • Memory Reassignment as a First-Class Citizen: Memory grants and steals are presented not just as niche recovery mechanisms, but as fundamental tools for efficient data movement and dynamic resource management.
  • Failure Independence is an Opportunity: The fact that compute and memory can fail independently is framed as a feature to be exploited for faster recovery and resilience, rather than just a problem to be managed.
  • Co-design is Key: The authors advocate for a co-design approach between applications and disaggregated operating systems, suggesting that this synergy is essential for realizing the full potential of DDCs.
  • Inspiration from Legacy Systems: The paper draws parallels to early distributed shared memory systems, noting how initial transparency goals often gave way to more performance-oriented, less transparent models.

Key Results

The paper presents case studies demonstrating the potential benefits:

  • Data Parallel Frameworks: Memory grants can eliminate unnecessary data movement during shuffles, reducing the number of network round-trips from three to one, leading to faster data transfer.
  • Fault-Tolerant Applications (Paxos): Memory steals can enable faster reincarnation of failed Paxos replicas by allowing a new replica to directly take ownership of the failed replica's memory, bypassing the need to fetch state from checkpoints or other replicas. Failure notifications can also speed up the detection and handling of memory failures.

Limitations and Future Work

The authors acknowledge that their work is largely speculative and poses more questions than it answers. They highlight several challenges:

  • Interconnect Flexibility: Implementing memory grants and steals requires an interconnect that can handle dynamic remapping and fine-grained control.
  • Crash Consistency: The "memory steal" operation introduces challenges in maintaining crash consistency, as volatile state can now transcend application instances. Applications need robust programming models to ensure semantic coherence.
  • Namespace and Metadata Management: When memory is reassigned, applications need mechanisms to locate their data, as traditional compiler-based tracking might not suffice. The paper suggests using file-system-like namespaces or distributing metadata.
  • Feasibility of Rack MMU: While programmable switches offer some capabilities, achieving the required combination of low latency, programmability, and reconfigurability for a Rack MMU remains an open problem.
  • Application Adoption: The proposed interfaces represent a significant departure from existing programming models, and their adoption will depend on clear benefits and developer familiarity.

Future work could involve implementing these primitives in a real OS, developing more sophisticated programming models for memory reassignment, and exploring their application to a wider range of distributed systems.

Summary updated

Abstract

This paper examines disaggregated data center architectures from the perspective of the applications that would run on these data centers, and challenges the abstractions that have been proposed to date. In particular, we argue that operating systems for disaggregated data centers should not abstract disaggregated hardware resources, such as memory, compute, and storage away from applications, but should instead give them information about, and control over, these resources. To this end, we propose additional OS abstractions and interfaces for disaggregation and show how they can improve data transfer in data parallel frameworks and speed up failure recovery in replicated, fault-tolerant applications. This paper studies the technical challenges in providing applications with this additional functionality and advances several preliminary proposals to overcome these challenges.
BibTeX
@inproceedings{angel2020disaggregation,
  author = {Sebastian Angel and Mihir Nanavati and Siddhartha Sen},
  title = {{Disaggregation and the Application}},
  booktitle = {HotCloud},
  year = {2020},
  url = {https://openreview.net/forum?id=EQPAFTyoT0},
}
Figure