Content-Based Dense Retrieval of Open Datasets

Qiaosheng ChenDC@ISWC 2024 · OpenReview

Introduction

As the volume of open data continues to expand across various domains, the ability to efficiently locate relevant datasets has become a critical challenge. Current dataset search engines, such as Google Dataset Search, rely heavily on human-provided metadata—titles, tags, and descriptions. However, this metadata is frequently incomplete, inaccurate, or entirely absent, rendering vast amounts of valuable data "invisible." This research proposes a paradigm shift toward content-based dense retrieval, leveraging pre-trained language models to index and search the actual data contained within files.

The Metadata Dependency Problem

The reliance on metadata creates a significant bottleneck in information retrieval. If a researcher fails to provide a comprehensive description for a CSV or RDF file, traditional keyword-based search engines cannot index the underlying information. Three primary obstacles hinder the transition to content-based search:

  1. Scale and Token Limits: Datasets often exceed the input constraints of standard transformer models (e.g., BERT).
  2. Format Heterogeneity: Open data exists in diverse structures, including tables (CSV), graphs (RDF), and hierarchical formats (JSON).
  3. Lack of Specialized Benchmarks: Most Information Retrieval (IR) test collections are optimized for unstructured text documents rather than structured datasets.

Methodology: Content-Based Retrieval

To address these challenges, the proposed framework introduces several innovations designed to bridge the gap between user queries and raw data.

Unified Triple Representation

To manage heterogeneity, the framework transforms various data formats into a unified Subject-Predicate-Object (triple) format. This abstraction allows a single model to process diverse data types—such as a row in a spreadsheet or a node in a knowledge graph—using a consistent logical structure.

Data Summarization Strategies

Given the massive size of many datasets, full-text indexing is often computationally prohibitive. The research explores two summarization paths:

  • Static Summarization: Utilizing tools like IlluSnip and PCSG to extract the most representative "snippets" or structural cores of a dataset.
  • Dynamic Summarization: Selecting data snippets in real-time based on their specific relevance to a user’s unique query, ensuring that the most pertinent content is prioritized during the retrieval phase.

Coarse-to-Fine Tuning

Training dense retrieval models requires substantial labeled data, which is currently scarce for dataset search. The author proposes a two-stage training pipeline:

  1. Coarse-Tuning: Using weak supervision, such as treating dataset titles as proxy queries, to initialize the model.
  2. Fine-Tuning: Refining the model on high-quality, human-annotated datasets to improve precision and semantic alignment.

Benchmarking and Evaluation

A core contribution of this work is the development of the ACORDAR and ACORDAR 2.0 test collections. ACORDAR 2.0 specifically addresses the limitations of keyword-based benchmarks by using Large Language Models (LLMs) to generate natural, "question-style" queries that better reflect real-world information-seeking behavior. Preliminary evaluations focus on metrics such as NDCG (Normalized Discounted Cumulative Gain) and MAP (Mean Average Precision), with the expectation that content-enhanced models will significantly outperform traditional sparse retrieval methods like BM25.

Conclusion and Future Directions

While content-based dense retrieval offers a promising path forward, it introduces new challenges regarding computational intensity and storage requirements. Future work will focus on integrating LLMs directly into the search pipeline to enhance query understanding and expanding these techniques to support multi-modal retrieval, encompassing images, video, and audio datasets. By treating the data itself as the primary source of truth, we can move toward a more robust and transparent ecosystem for open data discovery.

Summary updated

BibTeX
@inproceedings{chen2024content,
  author = {Qiaosheng Chen},
  title = {{Content-Based Dense Retrieval of Open Datasets}},
  booktitle = {DC@ISWC},
  year = {2024},
  url = {https://openreview.net/forum?id=P2EZCGpJPq},
}
Figure