Constructing an OLAP cube from distributed XML data

Tapio Niemi, Marko Niinimäki, Jyrki Nummenmaa, Peter ThanischDOLAP 2002 · DOI

TL;DR

This paper introduces a novel system for constructing On-Line Analytical Processing (OLAP) cubes from distributed, heterogeneous data sources using XML. Instead of maintaining a constantly updated, massive OLAP database, this approach builds cubes on demand, tailored to specific user queries expressed in MDX, leading to more efficient data analysis.

What Problem Does This Paper Address?

Traditional OLAP systems rely on large, pre-built data warehouses. However, as data sources grow and user information needs become more dynamic and unpredictable, maintaining these warehouses becomes a significant challenge. It's difficult to anticipate all potential queries, leading to either over-provisioning (storing vast amounts of data that might never be used) or under-provisioning (missing crucial data for specific analyses). Furthermore, data often resides in heterogeneous repositories, making integration complex. This paper addresses the need for a more flexible and efficient approach to OLAP by enabling the on-demand construction of smaller, targeted OLAP cubes from distributed data.

Methods

The core of the proposed system lies in its ability to construct OLAP cubes dynamically based on user queries. Here's a breakdown of the key methods:

  • XML for Data Representation: The system leverages XML to represent both the OLAP cube schema and the data itself. This choice is driven by XML's flexibility in handling heterogeneous data and its widespread adoption. The paper proposes a "star schema" style XML formalism, which includes a fact table (containing measures and dimension keys) and dimension tables (holding hierarchy information), offering a more economical storage and easier management of dimension data.
  • MDX for Querying: Users interact with a virtual "universal" OLAP cube schema using the MultiDimensional Expressions (MDX) query language. This familiar interface allows users to express their analytical needs without needing to know the underlying data distribution or structure.
  • Query Analysis and Data Collection: The system's "Collection Server" parses the MDX query to identify the required measures, dimensions, and constraints. Based on this analysis and a distribution model (which specifies where different parts of the data reside), the server requests the necessary data from distributed data warehouses.
  • On-Demand Cube Construction: Data retrieved from various sources is collected in XML format. The Collection Server then performs necessary data transformations, aggregations, and selections to construct a tailored OLAP cube. This cube is then loaded into an OLAP server for fast query answering.
  • Spitfire and Java Implementation: The prototype system is implemented using Java, with Spitfire software facilitating access to relational databases via HTTP/HTTPS. Spitfire converts relational data into XML, which is then processed by the Collection Server.

Datasets

The paper uses a real-world example of "world trade data" to illustrate the system. This dataset consists of millions of entries detailing pairwise import/export figures for numerous countries over several years, classified by product groups. The data is distributed across three different databases. No new benchmarks are introduced; the focus is on demonstrating the system's capability with existing, albeit large, distributed data.

Key Insights and Ideas

  • On-Demand Cube Construction is Feasible and Beneficial: The paper argues that building OLAP cubes only when needed, rather than maintaining a static, massive warehouse, is a more practical and efficient approach for many applications, especially with dynamic data and evolving user needs.
  • XML as a Unifying Data Format: Using XML as an intermediate format for data collection and representation effectively bridges the gap between heterogeneous data sources and the OLAP server.
  • MDX as a User-Friendly Interface for Data Definition: Allowing users to define cube contents via familiar MDX queries simplifies the process of specifying analytical requirements, abstracting away the complexities of data distribution and integration.
  • "Universal" Schema Simplifies User Interaction: Presenting a single, conceptual "universal" schema to the user, regardless of the underlying distributed nature of the data, significantly improves usability.

Key Results

The paper demonstrates the feasibility of its approach through a prototype implementation using world trade data. While specific quantitative performance metrics (like query speedup percentages) are not detailed, the core result is the successful construction of an OLAP cube from distributed data sources based on an MDX query. The system can analyze an MDX query, determine the required data, collect it from distributed repositories in XML, and load it into an OLAP server. The authors emphasize that while cube construction itself is not "on-line," the subsequent queries against the constructed cube are expected to be significantly faster due to the targeted nature of the cube.

Limitations and Future Work

  • Aggregation Distribution: The authors acknowledge that distributing aggregation and selection processes to the data sources themselves would further enhance efficiency. Currently, the collection server handles these tasks.
  • Vertical Distribution: The paper focuses on horizontal distribution of data and does not delve deeply into vertical distribution strategies.
  • XML Transformation Complexity: While XML and XSLT are powerful, managing complex transformations for highly heterogeneous data sources can be challenging.
  • Future Work: The authors suggest exploring distributed aggregation and selection mechanisms. They also mention the potential for using standard XML tools like XSLT for data manipulation and aggregation, and investigating APIs of OLAP systems for more direct data loading, bypassing intermediate files.

Summary updated

BibTeX
@inproceedings{niemi2002constructing,
  author = {Tapio Niemi and Marko Niinimäki and Jyrki Nummenmaa and Peter Thanisch},
  title = {{Constructing an OLAP cube from distributed XML data}},
  booktitle = {DOLAP},
  year = {2002},
  doi = {10.1145/583890.583894},
}
Figure