NYUCIN at the NTCIR-16 Dataset Search 2 Task

Levy de Souza Silva, Luciano Barbosa, Sonia Castelo, Haoxiang Zhang, Aécio S. R. Santos, Juliana FreireNTCIR 2022 · OpenReview

Abstract

The NYUCIN team secured first place in the NTCIR-16 Dataset Search 2 challenge by implementing a two-stage retrieval pipeline enhanced by a novel transfer learning strategy. By pre-training a hybrid deep learning matching model on a large-scale news-to-table alignment task, the team addressed the scarcity of labeled dataset search data, achieving a 30% improvement in nDCG@5 over the next best competitor.

Introduction

While web search is a mature field, dataset search—finding specific statistical or scientific data—remains a significant challenge. Unlike standard web pages, datasets are composed of complex metadata (titles, descriptions), headers, and rows of structured values. The primary hurdle for researchers is the "cold start" problem: there is very little labeled training data (queries paired with "correct" datasets) to teach machine learning models how to rank results effectively. The NTCIR-16 task provided only 141 highly relevant query-dataset pairs, which is insufficient to train a modern deep neural network from scratch.

Methodology

The authors proposed a two-stage retrieval pipeline consisting of candidate retrieval and deep re-ranking.

Candidate Retrieval

Initial retrieval was performed using Elasticsearch with the BM25 algorithm to extract the top 100 candidate datasets from the corpus. Empirical results indicated that searching against the dataset Description field yielded significantly better recall than searching the Title alone, as titles in scientific datasets are often cryptic or overly brief.

The Matching Model Architecture

To re-rank the candidates, the team developed a hybrid deep learning architecture designed to score the similarity between queries and metadata. As shown in Figure 1, the model features two parallel branches:

  • The Attention Branch: Utilizes FastText embeddings and a Bidirectional GRU to capture context, followed by a cross-attention layer to identify interactions between query terms and dataset metadata.
  • The Transformer Branch: Employs a pre-trained BERT model to create a contextual representation of the query-dataset pair.
  • Fusion: The outputs of both branches are integrated into a Multi-Layer Perceptron (MLP) to predict a final similarity score.
Figure 1: The architecture of the NYUCIN matching model, featuring parallel Attention and Transformer branches.

Transfer Learning via Distant Supervision

To overcome the data scarcity, the team utilized distant supervision. They pre-trained the model on a massive dataset of 84,000 pairs of news articles and web tables. The hypothesis was that the semantic logic required to match unstructured text (news) to structured data (tables) would transfer effectively to the task of matching search queries to datasets.

User Interface and Spatial Awareness

Beyond the retrieval model, the study explored how users interact with dataset search results through the Auctus platform. A key finding was the importance of spatial metadata. In user studies, 96.6% of participants relied on spatial coverage maps when answering geographic data questions. For data scientists, visualizing the geographic provenance of data is often as critical as the textual content.

Figure 2: The Auctus search interface, highlighting dataset results (B1) and spatial coverage visualization (B2).

Results

The NYUCIN approach attained the highest score among all submitted runs in the NTCIR-16 competition. Their model achieved an nDCG@5 of 0.246, representing a 30% improvement over the second-best result. Internal benchmarks showed the deep learning model was at least 37% more effective than standard baselines like the Universal Sentence Encoder (USE) or BM25 alone.

Conclusion

The success of the NYUCIN team demonstrates that cross-domain transfer learning can effectively mitigate the lack of training data in specialized search tasks. Future work involves integrating advanced data profiling to allow the search engine to understand relationships between data rows, potentially enabling users to navigate between related tables in a more interconnected data ecosystem.

Summary updated

BibTeX
@inproceedings{silva2022nyucin,
  author = {Levy de Souza Silva and Luciano Barbosa and Sonia Castelo and Haoxiang Zhang and Aécio S. R. Santos and Juliana Freire},
  title = {{NYUCIN at the NTCIR-16 Dataset Search 2 Task}},
  booktitle = {NTCIR},
  year = {2022},
  url = {https://openreview.net/forum?id=syKpOEFoqF},
}
Figure