TL;DR
This paper introduces TODS, a cluster object storage platform designed to simplify the development of scalable Internet services. TODS offers a unified, transparent, and object-oriented view of storage, supporting ACID transactions and transparent persistence for Java applications, aiming to abstract away complex data management challenges.
What Problem Does This Paper Address?
Building scalable Internet services on clusters presents significant challenges, particularly in data management. Existing solutions like RDBMS and distributed file systems often fall short for cluster environments. RDBMS can be overly complex and don't always align well with object-oriented programming, leading to an "impedance mismatch" where developers spend considerable effort converting data between application objects and database records. Distributed file systems, while offering distribution and fault tolerance, lack transaction support and leave data organization entirely to developers. This forces developers to often build custom, service-specific storage solutions, mixing application logic with low-level data management, which hinders portability and maintainability. TODS aims to bridge this gap by providing a dedicated, object-oriented, and transparent data management layer for cluster-based Internet services.
Methods
TODS employs a decentralized architecture with several key components:
- Peer Servers: These are the primary connection points for services, presenting a unified view of the entire TODS system. They abstract data distribution and act as transaction managers for distributed transactions.
- TODSLib: A Java library that sits within the service process, mapping user API calls to messages sent to Peer Servers. Crucially, it implements "transparent persistence," automatically handling the saving and loading of Java objects to and from the store without requiring developers to write explicit I/O code. This is achieved through on-the-fly bytecode enhancement of user classes.
- Bricks: These are single-node transactional data stores that manage local data using a key-value approach. They provide local transaction support and primitive query functionalities, built upon Berkeley DB.
- Meta-Server: This component maintains system configuration and metadata, including namespace structure, user information, and class definitions. It is replicated for fault tolerance.
The system supports both transactional and non-transactional access modes. For transactional access, it utilizes a two-phase commit protocol for distributed transactions and leverages the transactional capabilities of Bricks for local transactions. For non-transactional access, it relies on caching mechanisms to improve performance.
Datasets
Not discussed in the paper. The paper focuses on the design and performance of the TODS system itself, using its own internal data structures and simulated workloads for evaluation.
Key Insights and Ideas
- Transparent Persistence for Java: The core innovation is enabling Java applications to treat persistent objects as if they were in-memory objects. This significantly simplifies development by abstracting away the complexities of data serialization, storage, and retrieval.
- Object-Oriented Data Model for Clusters: The paper argues that an object-oriented data model is a better fit for Internet services than a relational model, offering better alignment with programming languages and enabling more granular data access optimizations.
- Decentralized Architecture for Scalability: The peer-to-peer nature of Peer Servers and the distributed storage across Bricks are designed to achieve high scalability and availability, leveraging the inherent properties of clusters.
- Separation of Concerns: TODS strictly separates data management from service logic, allowing developers to focus on application functionality rather than low-level storage details.
- Performance-Oriented Design: The system incorporates various caching strategies (transaction cache, object cache, buffer cache) and leverages fast cluster interconnects to achieve high throughput.
Key Results
- Scalability: The system demonstrates near-linear scalability, achieving 11,160 in-memory reads/sec on a 36-node cluster.
- Throughput:
- In-memory reads: Up to 11,160 reads/sec for 1KB objects and 1,332 reads/sec for 128KB objects.
- On-disk reads: Approximately 2740 reads/sec for 1KB objects, with payload bandwidth reaching 46MB/s for 128KB objects.
- Cache-hit reads: 13,392 reads/sec for 1KB objects and 1,566 reads/sec for 128KB objects, showing a significant improvement over on-disk reads.
- Transactional Performance: The system can handle 396 transactions/sec when all 36 Bricks participate in local transactions. Distributed transactions are about 1/3 the performance of local transactions.
- Ease of Use: A comparative experiment showed that developing a simple music jukebox service using TODS (via JDO) took half the time and resulted in fewer lines of code compared to using JDBC with RDBMS.
Limitations and Future Work
- Large Data Files: TODS currently handles large data files inefficiently, loading them entirely into memory. The authors suggest adding a stream-oriented access method.
- Garbage Collection: Persistent object garbage collection is not implemented, requiring manual deletion of unused objects.
- Schema Evolution: Support for schema evolution is missing, which is important for long-term data management.
- Transports: The authors suggest exploring additional transport protocols like VIA and local shared memory messaging.
- Inter-Peer Server Cache Cooperation: Enhancing Object Caches to cooperate between Peer Servers could further improve hit rates.
- Indexing: Bricks currently perform linear scans for queries; introducing indexing support is planned.
- Replication: Replication of user data is not yet implemented but is considered relatively easy to add.
Summary updated