Context Propagation in Conversational Search Utterances Participation of the CNR Team in CAsT 2022

Ida Mele, Cristina Ioana Muntean, Franco Maria Nardini, Raffaele Perego, Nicola TonellottoTREC 2022 · OpenReview

Introduction

In Conversational Information Seeking (CIS), users often rely on dialogue history, using pronouns or omitting context entirely in subsequent turns. For instance, a user might follow "Who is the CEO of Apple?" with "How old is he?". To effectively retrieve documents, a system must resolve these dependencies through utterance rewriting. The CNR team's participation in TREC CAsT 2022 focuses on "context propagation"—the process of injecting information from the conversation history into the current query to make it self-contained and searchable.

Methodology

The team implemented a two-stage pipeline consisting of utterance rewriting followed by a retrieval and re-ranking process.

Utterance Rewriting Strategies

Four distinct strategies were tested to enrich the raw user utterance ($u_i$):

  1. CNR-run1: Propagates topics (noun chunks extracted via SpaCy) from the automatically rewritten previous utterance provided by the CAsT organizers.
  2. CNR-run2: Propagates topics from the manually rewritten (human-corrected) previous utterance, serving as a high-quality baseline.
  3. CNR-run3: Enriches the query using the first sentence of the system's previous response.
  4. CNR-run4: Enriches the query using the top-5 most frequent terms from the system's previous response.

Retrieval and Re-ranking

The initial retrieval phase utilized the PyTerrier framework with a DPH (hypergeometric weighting) model for unsupervised sparse retrieval. To improve precision, the top 200 results were then processed by a BERT-based re-ranker fine-tuned on the MS MARCO dataset.

Experimental Setup

The evaluation used the TREC CAsT 2022 dataset, which features dialogue trees across a massive document corpus. This corpus includes approximately 5 million Wikipedia articles (KILT), 11.9 million MS MARCO V2 documents, and 728,000 news articles from the TREC Washington Post (V4) collection.

Results and Analysis

The performance was evaluated using Mean Average Precision (MAP) and Normalized Discounted Cumulative Gain (nDCG@20) under both "Lenient" and "Strict" relevance settings.

Initial retrieval performance across different context propagation strategies under Lenient and Strict settings.

The initial retrieval results (shown above) demonstrate the baseline effectiveness of each strategy. Notably, CNR-run2, which utilizes manual rewrites, consistently outperformed the other runs, highlighting the gap between current automated context resolution and human-level understanding. The table also includes track-wide averages (Min, Median, Max) for comparison.

Final performance metrics for the four runs after applying BERT-based re-ranking.

The application of the BERT re-ranker provided a significant performance boost across all metrics. For example, CNR-run2's nDCG@20 improved from 0.1832 to 0.2200. Interestingly, CNR-run3 (using the first sentence of the previous response) outperformed CNR-run1 (using automatic rewrites). This suggests that the system's own previous answer can be a more reliable context source than a potentially noisy automated rewrite of a previous query.

Conclusion

The CNR team's experiments demonstrate that while manual context remains the gold standard, leveraging the system's previous responses is a promising avenue for automatic query enrichment. Future work will focus on more sophisticated dependency modeling to better capture the specific relationships between dialogue turns and user intent.

Summary updated

BibTeX
@inproceedings{mele2022context,
  author = {Ida Mele and Cristina Ioana Muntean and Franco Maria Nardini and Raffaele Perego and Nicola Tonellotto},
  title = {{Context Propagation in Conversational Search Utterances Participation of the CNR Team in CAsT 2022}},
  booktitle = {TREC},
  year = {2022},
  url = {https://openreview.net/forum?id=mEEt9IqMOZ6},
}
Figure