Ledgerdata Refiner: A Powerful Ledger Data Query Platform for Hyperledger Fabric

Ence Zhou, Haoli Sun, Bingfeng Pi, Jun Sun, Kazuhiro Yamashita, Yoshihide Nomura

Abstract

Hyperledger Fabric is a leading permissioned blockchain designed for enterprise-grade applications. However, it faces significant data access limitations; ledger data can typically only be retrieved via restricted interfaces provided by SDKs or chaincode. This paper presents Ledgerdata Refiner, a middleware platform that overcomes these constraints by extracting ledger data into a relational database and automatically inferring schemas for state variables. This enables complex, SQL-like "rich queries" and historical state tracking without requiring changes to existing chaincode. Our evaluation on a real-world application demonstrates that the platform maintains high throughput, effectively decoupling analytical workloads from the core blockchain consensus process.

Introduction: The "Data Silo" Problem in Fabric

While Hyperledger Fabric is a robust choice for enterprise blockchain, it often suffers from a "data silo" problem. Unlike traditional databases, Fabric's native state database (often LevelDB) primarily supports basic key-value lookups. Performing complex queries—such as finding all assets belonging to a specific owner—is computationally expensive or impossible without specific indexing. Furthermore, tracking the historical evolution of an asset typically requires pre-configured indices that are difficult to access externally. For businesses requiring deep analytics or audit trails, these technical hurdles make Fabric data difficult to utilize for business intelligence.

System Architecture

To address these limitations, we propose Ledgerdata Refiner, an analysis middleware designed to bridge the gap between blockchain ledgers and flexible data querying. The system architecture consists of three core processes: synchronization, analysis, and browsing.

Architecture of the Ledgerdata Refiner platform, showing the flow from blockchain ledger data through analysis middleware to a reorganized database.
  1. Ledger Synchronization & Parsing: The system connects to a Fabric peer as a client, incrementally retrieves blocks, and parses complex transaction metadata (creators, endorsers, read-write sets, timestamps). This data is reorganized into a PostgreSQL relational database.
  2. State Schema Analysis: Since Fabric stores state as simple key-value pairs (often JSON), the system performs automated schema extraction to identify the structure and data types of these objects.
  3. Schema Clustering: To handle evolving data, the system compares new schemas against known ones using categories such as Equal, Contain, or Different. This allows the platform to group similar types of assets together, even if their internal structures vary slightly over time.

Data Synchronization and Schema Mapping

A critical feature of Ledgerdata Refiner is its ability to map unstructured or semi-structured ledger data into a structured format suitable for relational queries. By maintaining a clear correspondence between state keys and inferred schemas, the platform allows users to perform SQL-like queries against the "world state."

Mechanism for synchronizing ledger data to a third-party database and establishing schema correspondence.

This decoupling of analysis from consensus ensures that heavy analytical queries do not slow down the blockchain's primary job of processing transactions. Users can then use a web-based UI to perform complex filters on transactions or "rich queries" on the world state using the inferred schemas as a guide.

Evaluation and Results

The authors evaluated the platform using a custom-built Information Disclosure Platform for SMEs. The test involved submitting 100,000 transactions to a Fabric test network using four distinct JSON schemas ranging from simple to complex nested structures.

  • High Throughput: The synchronization and parsing process reached a throughput of approximately 1,400 Transactions Per Second (TPS).
  • Efficient Analysis: The schema analysis process maintained a steady rate of about 1,200 TPS.
  • Scalability: Processing time for both synchronization and schema analysis proved to be linear relative to the number of transactions, suggesting the platform can scale effectively for enterprise-sized ledgers.
  • Performance Match: The framework's throughput exceeds the typical performance of a standard Fabric network (~1,000 TPS), meaning the Refiner can keep up with a live network in real-time.

Conclusion and Future Work

Ledgerdata Refiner provides a robust solution for the data access limitations of Hyperledger Fabric. By automating schema inference and leveraging relational databases, it empowers users with rich query capabilities and historical tracking without modifying existing chaincode. While there is a small inherent delay in relational mapping, the benefits for business intelligence are substantial. Future work will explore even more efficient interfaces and deeper analytical tools to provide "sufficient and efficient" data access for end-users, potentially adapting the platform for more complex financial transaction formats.

Summary updated

Abstract

Blockchain is one of the most popular distributed ledger technologies. It can solve the trust issue among enterprises. Hyperledger Fabric is a permissioned blockchain aiming at enterprise-grade business applications. However, compared to traditional distributed database solutions, one issue of blockchain based application development is the limited data access. For Fabric, the ledger data can only be retrieved by limited interfaces provided by Fabric SDKs or chaincode. In order to meet the requirements of data query and provide flexible query functions for real applications built on Fabric, this paper proposed a ledger data query platform called Ledgerdata Refiner. With ledger data analysis middleware, we provide sufficient interfaces for users to retrieve block or transaction efficiently. It is also able to track historical operations for any specific state. In addition, schemas of ledger state have been analyzed and clustered, which enable users to perform rich queries against ledger data. Finally, we validate the effectiveness of our query platform on a real application.
BibTeX
@misc{zhou2019ledgerdata,
  author = {Ence Zhou and Haoli Sun and Bingfeng Pi and Jun Sun and Kazuhiro Yamashita and Yoshihide Nomura},
  title = {{Ledgerdata Refiner: A Powerful Ledger Data Query Platform for Hyperledger Fabric}},
  year = {2019},
  eprint = {1912.04526},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.04526},
}
Figure