A distributed persistent object store for scalable service

Chao Jin, Weimin Zheng, Feng Zhou, Yinghui WuACM Sigops Oper. Syst. rev 2002 · DOI

TL;DR

This paper introduces TODS (Tsinghua Object Data Store), a novel distributed persistent object store designed to simplify the development of scalable network services on clusters. TODS provides a unified, transparent, object-oriented view of storage across a cluster, supporting ACID transactions and offering a more developer-friendly interface than traditional file systems or RDBMS.

What Problem Does This Paper Address?

Building scalable and highly available network services on clusters presents a significant challenge for data management. Existing solutions like file systems lack crucial features such as transaction support, data recovery, and efficient querying, making them unsuitable for structured or semi-structured data common in network services. Relational database management systems (RDBMS), while offering ACID compliance, often introduce performance overheads (especially with SQL) that hinder high-throughput, low-latency network applications. Distributed file systems, while addressing some distributed data management complexities, are often designed for wide-area networks and don't fully leverage the capabilities of modern high-speed clusters. This leaves a gap for a storage system that is scalable, highly available, performant, and simplifies the development of transactional services on clusters.

Methods

TODS employs a decentralized, peer-server architecture to achieve scalability and availability. Key components include:

  • TODSLib: A client-side library that provides a transparent persistence interface, compatible with Java Data Objects (JDO). It allows Java applications to access objects stored in TODS without explicit I/O code, handling object fetching and persistence automatically.
  • Peer Servers: These act as agents for client applications, managing object caches, handling concurrency control, and coordinating distributed transactions using a two-phase commit protocol. They also facilitate distributed queries.
  • Bricks: These are the underlying single-node transactional data stores, implemented using Berkeley DB. They manage local data in a key-value fashion and provide local transaction and primitive query functionalities.
  • Meta-Server: Responsible for managing system configuration, including namespace structure, user information, and class metadata. It is replicated for high availability.

TODS utilizes an object-based data model where objects are stored in "Object Spaces." Each persistent object has a globally unique Object ID (OID), which encodes information about its location, class, and object space. The system supports both transactional and non-transactional access, with different consistency guarantees.

Datasets

Not discussed in the paper. The paper focuses on system design and performance evaluation using synthetic workloads.

Key Insights and Ideas

  • Transparent Persistence Simplifies Development: By abstracting away I/O and persistence logic into a JDO-compatible interface, TODS significantly reduces the burden on developers, allowing them to focus on service logic.
  • Object Granularity for Concurrency: Using objects as the unit of management, rather than database rows or file blocks, allows for finer-grained concurrency control, leading to higher throughput.
  • Decentralized Architecture for Scalability: The peer-server model avoids the single point of failure and performance bottleneck often seen in traditional client-server systems.
  • Layered Data Access for Flexibility and Efficiency: The separation of concerns between TODSLib, Peer Servers, and Bricks, with caching at multiple levels, provides a good balance between flexibility and performance.
  • OID as a Physical Identifier: Using a physical OID that directly indicates object location simplifies object lookup and caching compared to logical IDs or path-based approaches, especially in a well-managed cluster environment.

Key Results

The paper presents performance experiments on a 36-node cluster.

  • Scalability: The system scales smoothly, achieving 11,160 in-memory reads/sec on a 36-node cluster.
  • Throughput: For 1KB objects, the system can handle 11,160 reads/sec with a payload bandwidth of 11MB/s. For 128KB objects, it achieves 1,332 reads/sec with a payload bandwidth of 170MB/s.
  • Cache Effectiveness: Object cache hits provide a significant performance boost, showing ~20% more throughput than in-memory tests and 4 times higher throughput than on-disk tests.
  • Transactional Performance: The system can perform 396 transactions/sec when all 36 Bricks participate in local transactions. Distributed transactions incur an overhead, resulting in performance roughly 1/3 of local transactions.

Limitations and Future Work

The authors acknowledge several limitations and suggest future work:

  • Large Block Data Handling: TODS is not efficient for large block data (e.g., PDF, TAR archives); a stream-oriented access method is desirable.
  • Garbage Collection: Persistent object garbage collection is not implemented, requiring users to explicitly delete unused objects.
  • Schema Evolution: Schema evolution is not supported, which is crucial for long-term data management.
  • Transport Protocols: Exploring alternative transports like VIA and local shared memory messaging could improve performance.
  • Inter-Peer Server Caching: Enhancing cooperation between Object Caches in different Peer Servers could further improve hit rates.

Summary updated

BibTeX
@article{jin2002distributed,
  author = {Chao Jin and Weimin Zheng and Feng Zhou and Yinghui Wu},
  title = {{A distributed persistent object store for scalable service}},
  journal = {ACM Sigops Oper. Syst. rev},
  year = {2002},
  doi = {10.1145/583800.583805},
}
Figure