Neural Language Model Based Attentive Term Dependence Model for Verbose Query (Student Abstract)

Dipannita Podder, Jiaul H. Paik, Pabitra MitraAAAI 2023 · DOI

Abstract

Verbose queries—long, conversational search strings—present a significant challenge in Information Retrieval (IR) due to the presence of non-informative terms. This paper introduces the BERT-based Attentive Term Dependence Model (B-ASD), an unsupervised approach that integrates neural attention weights into a Markov Random Field (MRF) framework. By leveraging BERT to identify term importance through contextual correlation and recursive importance formulas, B-ASD achieves state-of-the-art ranking performance on large-scale datasets without the need for supervised training.

Introduction

In modern search environments, users often employ "verbose queries" that resemble natural language sentences rather than concise keyword lists. For example, a query like "What are the specific environmental impacts of lithium mining in South America?" contains several terms ("specific", "of", "in") that may dilute the search engine's focus. Traditional models, such as the Sequential Dependence (SD) model, attempt to capture relationships between neighboring words but often fail to distinguish between core concepts and linguistic noise. While neural retrieval models offer potential solutions, they typically require extensive labeled data and high computational resources. B-ASD addresses these issues by combining the structural reliability of classical IR with the semantic depth of pre-trained language models.

Methodology: The B-ASD Model

The BERT-based Attentive Term Dependence Model (B-ASD) is a hybrid framework designed to weight query terms based on their contextual significance.

1. Markov Random Field Foundation

The model builds upon the MRF-based SD model, which ranks documents by evaluating three types of cliques: single terms (unigrams), ordered phrases (bigrams), and unordered word groups.

2. Neural Attention and Contextual Correlation

Rather than assigning uniform weights, B-ASD utilizes a pre-trained BERT model to generate dense vector representations for the query and its constituent terms. The importance of a term is determined by its alignment with the overall query vector.

3. Recursive Importance and Regulated IDF

A key innovation is the recursive definition of term importance: a term is considered vital if it correlates strongly with the query discourse and relates closely to other important terms. This importance is further refined by a "regulated Inverse Document Frequency" (IDF) factor, ensuring the model prioritizes terms that are both contextually central and discriminative within the document collection.

4. Unsupervised Ranking

B-ASD operates in an unsupervised manner, calculating attention weights and ranking documents in a single pass. This avoids the need for expensive query-document pair labeling while maintaining high efficiency.

Experimental Evaluation

The model was evaluated using the description fields of topics from two standard web collections:

  • GOV2: Approximately 25 million documents from .gov domains.
  • ClueWeb09B: Approximately 50 million web documents.

Key Results

B-ASD was compared against several baselines, including the standard SD model, DSW, and the supervised DEEPTR-SD model.

Performance comparison of B-ASD against baseline models on GOV2 and ClueWeb09B datasets.

As shown in the results table, B-ASD consistently outperforms both unsupervised and supervised baselines. On the GOV2 dataset, it achieved an NDCG@10 of 0.478, surpassing the supervised DEEPTR-SD (0.470). Similar trends were observed on ClueWeb09B, where B-ASD reached an NDCG@10 of 0.211. The improvements across NDCG and ERR metrics highlight the model's ability to surface more relevant documents at the top of the search results.

Discussion and Future Work

The success of B-ASD demonstrates the efficacy of bridging classical probabilistic models with modern transformer-based architectures. By using BERT to provide weights for an MRF model, the system gains semantic depth without sacrificing the ranking efficiency of term-based models.

However, certain areas remain for future exploration. The computational overhead of generating BERT embeddings for every query term, while manageable in a single-pass architecture, warrants further latency analysis for real-time applications. Additionally, the sensitivity of the model to its empirical hyper-parameters ($d$ and $c$) and the potential for incorporating document-side contextual information are promising avenues for future research.

Summary updated

BibTeX
@inproceedings{podder2023neural,
  author = {Dipannita Podder and Jiaul H. Paik and Pabitra Mitra},
  title = {{Neural Language Model Based Attentive Term Dependence Model for Verbose Query (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.27010},
}
Figure