DRust: Language-Guided Distributed Shared Memory with Fine Granularity, Full Transparency, and Ultra Efficiency

Haoran Ma, Yifan Qiao, Shi Liu, Shan Yu, Yuanjiang Ni, Qingda Lu, Jiesheng Wu, Yiying Zhang, Miryung Kim, Harry Xu

TL;DR

This paper introduces DRust, a novel distributed shared memory (DSM) system that leverages Rust's ownership system to achieve fine-grained memory access, full transparency, and ultra-efficiency. By exposing Rust's ownership semantics to the runtime, DRust significantly simplifies coherence protocols, outperforming existing DSM systems in throughput and scalability.

What Problem Does This Paper Address?

Distributed Shared Memory (DSM) systems aim to provide a unified memory view across multiple machines, simplifying distributed application development. However, a major hurdle has been the complexity and overhead of maintaining memory coherence across servers. Existing DSM systems often rely on intricate protocols that necessitate extensive synchronization, leading to performance bottlenecks and poor scalability, especially compared to single-machine performance. This paper tackles the challenge of building a practical, high-performance DSM system that can overcome these limitations.

Methods

DRust's core innovation lies in its deep integration with the Rust programming language and its ownership system. Instead of traditional DSM approaches, DRust leverages Rust's compile-time guarantees of memory safety and thread safety, particularly its "single-writer-multiple-reader" (SWMR) property.

Key techniques include:

  • Language-Guided Coherence: DRust exposes Rust's ownership semantics to its runtime. This allows the system to understand data access patterns (e.g., exclusive writes vs. shared reads) directly from the program's structure, enabling a significantly simplified and more efficient coherence protocol.
  • Partitioned Global Address Space (PGAS): DRust implements a global heap distributed across servers, with each server managing a partition. Objects are assigned unique global addresses, allowing access from any server.
  • Ownership-Based Cache Coherence: DRust crafts a coherence protocol that uses Rust's ownership model. When a write occurs, the object is moved to the requesting server, and its address change implicitly invalidates cached copies on other servers. Reads are handled by fetching copies, and DRust ensures consistency through the ownership model.
  • Pointer Augmentation: DRust modifies Rust's pointer and reference types (Box, &mut, &) to carry additional information for the coherence protocol, such as local cache addresses and "color" bits for optimizing local writes.
  • Transparent Library Adaptation: DRust provides distributed versions of standard Rust libraries (threading, channels, mutexes) that maintain the same interfaces, allowing existing Rust programs to run distributively with minimal or no code changes.
  • Affinity Annotations: Optional annotations (TBox, spawn_to) allow developers to express data-to-computation affinity, further optimizing performance by co-locating data and threads.

Datasets

The paper evaluates DRust using four representative real-world applications:

  • DataFrame: An in-memory data analytics framework (similar to Spark/Pandas).
  • SocialNet: A microservice-based application representing a Twitter-like web service.
  • GEMM: General matrix multiplication, a compute-intensive task.
  • KV Store: An in-memory key-value cache engine (similar to Memcached).

These applications were chosen to cover a range of use cases and resource demands. The paper does not introduce new benchmarks but uses existing applications to demonstrate DRust's performance against established DSM systems.

Key Insights and Ideas

  • Rust's Ownership is a Game-Changer for DSM: The most significant insight is that Rust's compile-time ownership system, designed for single-machine safety, can be effectively leveraged to drastically simplify and accelerate distributed shared memory systems. This moves beyond traditional DSM approaches that often struggle with complex coherence protocols.
  • Semantic Information is Key: By understanding the semantics of data access (e.g., exclusive write vs. shared read) directly from the language, DRust avoids the need for generic, heavy-handed synchronization mechanisms common in other DSMs.
  • Fine-Grained Access is Achievable and Efficient: DRust demonstrates that fine-grained, object-level distributed access is not only possible but can be highly efficient when guided by language semantics, challenging the notion that DSM must be coarse-grained.
  • Transparency is Achievable with Library Adaptation: By adapting standard Rust libraries, DRust achieves a high degree of transparency, allowing existing Rust applications to run distributively with minimal or no code modifications, lowering the barrier to adoption.
  • Pointer "Coloring" Optimizes Local Writes: The use of "color" bits in pointers, inspired by garbage collectors, is a clever technique to efficiently handle local writes without incurring the full cost of moving the object every time, significantly improving performance in certain scenarios.

Key Results

DRust demonstrates significant performance improvements over state-of-the-art DSM systems like GAM and Grappa:

  • Throughput: DRust achieves up to 2.64x and 29.16x higher throughput than GAM and Grappa, respectively, across various applications.
  • Scalability: DRust scales much better with the number of nodes, showing substantial speedups compared to the original single-node implementations. For example, DataFrame achieved 5.57x higher throughput on eight nodes with DRust, compared to 2.18x for GAM and 1.69x for Grappa.
  • Overhead: DRust incurs a minimal overhead of only 2.42% compared to the original Rust program running on a single machine, indicating its efficiency.
  • Cache Coherence Cost: DRust's cache coherence overhead is significantly lower than GAM and Grappa, ranging from 4% to 32% in the worst case, compared to 10% to 98% for the baselines.
  • Thread Migration: DRust can migrate threads with an average latency of 218µs, enabling effective load balancing.

Limitations and Future Work

The authors acknowledge several limitations:

  • Unsafe Code: DRust's consistency guarantees only apply to "safe" Rust code. Developers using unsafe blocks are responsible for ensuring their own consistency.
  • Shared State Degeneration: In scenarios heavily reliant on shared states (like mutexes), DRust may degenerate into a traditional DSM system where accesses are serialized by a single server, limiting its advantages. However, the authors note that such patterns are less idiomatic in Rust.
  • ASLR Support: The current implementation does not support Address Space Layout Randomization (ASLR), though the design is compatible with it and future work will integrate this feature.

Future work could involve further exploring the benefits of affinity annotations, enhancing support for unsafe code, and integrating ASLR. The paper also suggests that DRust can benefit from advances in hardware-accelerated DSM technologies.

Summary updated

BibTeX
@inproceedings{ma2024drust,
  author = {Haoran Ma and Yifan Qiao and Shi Liu and Shan Yu and Yuanjiang Ni and Qingda Lu and Jiesheng Wu and Yiying Zhang and Miryung Kim and Harry Xu},
  title = {{DRust: Language-Guided Distributed Shared Memory with Fine Granularity, Full Transparency, and Ultra Efficiency}},
  booktitle = {OSDI},
  year = {2024},
  url = {https://openreview.net/forum?id=7Pjj04BbK0},
}
Figure