TL;DR
This paper introduces TODS (Tsinghua Object Data Store), a novel distributed persistent object store designed to simplify the development of scalable network services on clusters. TODS provides a unified, transparent, object-oriented view of storage across a cluster, supporting ACID transactions and offering a more developer-friendly interface than traditional file systems or RDBMS.
What Problem Does This Paper Address?
Building scalable and highly available network services on clusters presents a significant challenge for data management. Existing solutions like file systems lack crucial features such as transaction support, data recovery, and efficient querying, making them unsuitable for structured or semi-structured data common in network services. Relational database management systems (RDBMS), while offering ACID compliance, often introduce performance overheads (especially with SQL) that hinder high-throughput, low-latency network applications. Distributed file systems, while addressing some distributed data management complexities, are often designed for wide-area networks and don't fully leverage the capabilities of modern high-speed clusters. This leaves a gap for a storage system that is scalable, highly available, performant, and simplifies the development of transactional services on clusters.
Methods
TODS employs a decentralized, peer-server architecture to achieve scalability and availability. Key components include:
- TODSLib: A client-side library that provides a transparent persistence interface, compatible with Java Data Objects (JDO). It allows Java applications to access objects stored in TODS without explicit I/O code, handling object fetching and persistence automatically.
- Peer Servers: These act as agents for client applications, managing object caches, handling concurrency control, and coordinating distributed transactions using a two-phase commit protocol. They also facilitate distributed queries.
- Bricks: These are the underlying single-node transactional data stores, implemented using Berkeley DB. They manage local data in a key-value fashion and provide local transaction and primitive query functionalities.
- Meta-Server: Responsible for managing system configuration, including namespace structure, user information, and class metadata. It is replicated for high availability.
TODS utilizes an object-based data model where objects are stored in "Object Spaces." Each persistent object has a globally unique Object ID (OID), which encodes information about its location, class, and object space. The system supports both transactional and non-transactional access, with different consistency guarantees.
Datasets
Not discussed in the paper. The paper focuses on system design and performance evaluation using synthetic workloads.
Key Insights and Ideas
- Transparent Persistence Simplifies Development: By abstracting away I/O and persistence logic into a JDO-compatible interface, TODS significantly reduces the burden on developers, allowing them to focus on service logic.
- Object Granularity for Concurrency: Using objects as the unit of management, rather than database rows or file blocks, allows for finer-grained concurrency control, leading to higher throughput.
- Decentralized Architecture for Scalability: The peer-server model avoids the single point of failure and performance bottleneck often seen in traditional client-server systems.
- Layered Data Access for Flexibility and Efficiency: The separation of concerns between TODSLib, Peer Servers, and Bricks, with caching at multiple levels, provides a good balance between flexibility and performance.
- OID as a Physical Identifier: Using a physical OID that directly indicates object location simplifies object lookup and caching compared to logical IDs or path-based approaches, especially in a well-managed cluster environment.
Key Results
The paper presents performance experiments on a 36-node cluster.
- Scalability: The system scales smoothly, achieving 11,160 in-memory reads/sec on a 36-node cluster.
- Throughput: For 1KB objects, the system can handle 11,160 reads/sec with a payload bandwidth of 11MB/s. For 128KB objects, it achieves 1,332 reads/sec with a payload bandwidth of 170MB/s.
- Cache Effectiveness: Object cache hits provide a significant performance boost, showing ~20% more throughput than in-memory tests and 4 times higher throughput than on-disk tests.
- Transactional Performance: The system can perform 396 transactions/sec when all 36 Bricks participate in local transactions. Distributed transactions incur an overhead, resulting in performance roughly 1/3 of local transactions.
Limitations and Future Work
The authors acknowledge several limitations and suggest future work:
- Large Block Data Handling: TODS is not efficient for large block data (e.g., PDF, TAR archives); a stream-oriented access method is desirable.
- Garbage Collection: Persistent object garbage collection is not implemented, requiring users to explicitly delete unused objects.
- Schema Evolution: Schema evolution is not supported, which is crucial for long-term data management.
- Transport Protocols: Exploring alternative transports like VIA and local shared memory messaging could improve performance.
- Inter-Peer Server Caching: Enhancing cooperation between Object Caches in different Peer Servers could further improve hit rates.
Summary updated