Abstract
In an information-based world, there needs the support of big data. But the emergence of large data makes the various shortcomings of RDBMS exposed. The application of distributed database HBase can solve this problem. Based on HBase, a very popular distributed database, this paper proposes a method to migrate database which is on the huge amounts of data from RDBMS to HBase and demonstrates the realization of this semi-automatic migration design method in the process of data migration.
Introduction
The explosion of Web 2.0 applications and big data has exposed the scaling limitations of traditional relational database management systems (RDBMS). When dealing with massive, highly concurrent read/write operations, an RDBMS often struggles to maintain performance. NoSQL databases, specifically column-oriented stores like HBase, offer superior scalability, flexible data models, and high I/O utilization.
However, migrating legacy data from an RDBMS to HBase is complex because their underlying data models are fundamentally different. Most notably, HBase does not support SQL-style JOIN operations. This research proposes a semi-automatic method to translate relational schemas into HBase's multidimensional map structure without losing implicit relationships, aiming to reduce the manual engineering cost of database migration.
Methodology and Conversion Strategies
The proposed migration system consists of four main modules: an RDBMS manager, a table schema convertor, a schema adapter, and an HBase table manager. The core technical contribution is the Table Schema Convertor, which employs four distinct strategies to denormalize relational tables into HBase's column-family structure:
- Basic Convert: A direct 1:1 mapping where the RDBMS table name becomes the HBase table and column family name.
- Nest Convert: Used for 1:1, 1:N, or N:M relationships. It merges referenced tables into the referencing table by appending their columns, effectively pre-computing joins.
- Split Convert: The reverse of Nest Convert, appends columns from a shared referenced table into multiple referencing tables.
- Inline Convert: Handles transitive relationships by recursively applying Nest Convert logic to flatten hierarchies into a single table.
XML-Based Schema Mapping
The system workflow extracts the source schema (e.g., MySQL) via JDBC and represents it in an intermediary XML format. This decoupling allows for modular conversion rules and easier debugging.
After applying the conversion strategies, a target HBase schema is generated, also represented in XML. This file defines the row keys and column families necessary for the HBase environment, ensuring that relational associations are resolved during the design phase.
Key Insights and Results
- Pre-computed Joins: Since HBase lacks native
JOINsupport, relational associations must be resolved during the schema design phase. Denormalization ensures fast read access by storing associated records in a single row. - XML as an Intermediary: By using XML files to represent the original schema and the final HBase schema, the system decouples extraction from loading, making the tool more modular.
- Semi-Automation: While human oversight is needed to select specific conversion strategies (e.g., Nest vs. Split), the software automates the heavy lifting of schema redesign, significantly reducing manual overhead.
Conclusion and Future Work
This research successfully outlines a methodological framework for semi-automatic RDBMS-to-HBase migration. The use of XML-based mapping provides a flexible way to bridge the gap between relational and column-oriented models. Future work should focus on fully automating relationship detection, supporting a wider variety of source RDBMS platforms, and providing quantitative performance benchmarks to validate the efficiency of the migrated schemas.
Summary updated