Abstract
Modern enterprise data lakes often evolve into "data swamps," where finding relevant information among thousands of heterogeneous datasets is a significant challenge. This paper presents an ontology-driven search tool developed by IBM researchers to bridge the semantic gap between technical database schemas and business concepts. By mapping over 150,000 data fields to a curated ontology of 6,000 business terms, the system enables data scientists and analysts to discover assets using conceptual search and graph-based ranking. The tool is implemented as a microservices-based architecture on Kubernetes, providing a scalable solution for navigating massive enterprise data environments.
Introduction
Enterprise Data Lakes, such as IBM’s Cognitive Enterprise Data Platform (CEDP), are designed to centralize vast amounts of data to break down organizational silos. However, the sheer scale and heterogeneity of these environments often make data discovery nearly impossible for non-engineers. Data scientists frequently encounter cryptic table names and inconsistent field labeling (e.g., CUST_ID vs. CLIENT_REF), leading to a "semantic gap." Traditional data catalogs focus on governance and stewardship rather than user-centric discovery. This research addresses the need for a conceptual layer that allows users to search for data using the language of the business rather than the underlying database structure.
System Architecture and Methodology
The proposed discovery application is built on a microservices architecture deployed within a Kubernetes (k8s) cluster, interacting with the primary metadata catalog (IGC) and the CEDP Data Lake.
The system's core technical components include:
- Ontology Mapping: A unified ontology links 6,000 business terms to approximately 25,000 data assets and 150,000 fields.
- Metadata Ingestion Pipeline: A "Jena updater" service monitors a Kafka event stream from the governance catalog. An extractor service transforms metadata changes into RDF format, which is then stored in JenaDB (a triple store).
- Indexing and Search: The system utilizes Lucene for indexing business terms and SPARQL for querying the RDF representation.
- Semantic Search Features: To improve recall, the system employs an OWL reasoner for query expansion (e.g., mapping "email address" to "primary email"). Results are prioritized using graph network centrality, which ranks datasets based on their connectivity and relevance within the ontology.
Implementation and Scale
The tool was deployed on IBM’s internal CEDP, which encompasses over 300 data sources. The environment includes approximately 25,000 relational tables and over 150,000 individual data fields. This real-world implementation demonstrates the system's ability to handle enterprise-scale metadata while providing an intuitive interface for diverse user personas, including data scientists and business analysts.
Key Insights and Results
The research highlights the effectiveness of using an ontology as an abstraction layer. By focusing on a conceptual metadata layer rather than the raw data, the search engine can span heterogeneous sources without requiring a shared schema. Furthermore, graph-based ranking ensures that "well-connected" and high-value datasets are surfaced more prominently.
Qualitative evaluation through user surveys and Net Promoter Scores (NPS) indicates that the tool has significantly reduced the time required for data discovery within the IBM community. The microservices design ensures system resilience, allowing search functionality to remain available even during background metadata ingestion or updates.
Conclusion and Future Work
The ontology-driven discovery tool successfully addresses the "data swamp" problem by providing a semantic bridge for enterprise data lakes. Future developments will focus on integrating natural language processing (NLP) for more conversational queries, incorporating data quality metrics into search rankings, and extending the tool to support federated catalogs across multiple platforms. While current results are primarily qualitative, future work will also include quantitative benchmarking of "time-to-find" metrics.
Summary updated