Introduction
In Conversational Information Seeking (CIS), users often rely on dialogue history, using pronouns or omitting context entirely in subsequent turns. For instance, a user might follow "Who is the CEO of Apple?" with "How old is he?". To effectively retrieve documents, a system must resolve these dependencies through utterance rewriting. The CNR team's participation in TREC CAsT 2022 focuses on "context propagation"—the process of injecting information from the conversation history into the current query to make it self-contained and searchable.
Methodology
The team implemented a two-stage pipeline consisting of utterance rewriting followed by a retrieval and re-ranking process.
Utterance Rewriting Strategies
Four distinct strategies were tested to enrich the raw user utterance ($u_i$):
- CNR-run1: Propagates topics (noun chunks extracted via SpaCy) from the automatically rewritten previous utterance provided by the CAsT organizers.
- CNR-run2: Propagates topics from the manually rewritten (human-corrected) previous utterance, serving as a high-quality baseline.
- CNR-run3: Enriches the query using the first sentence of the system's previous response.
- CNR-run4: Enriches the query using the top-5 most frequent terms from the system's previous response.
Retrieval and Re-ranking
The initial retrieval phase utilized the PyTerrier framework with a DPH (hypergeometric weighting) model for unsupervised sparse retrieval. To improve precision, the top 200 results were then processed by a BERT-based re-ranker fine-tuned on the MS MARCO dataset.
Experimental Setup
The evaluation used the TREC CAsT 2022 dataset, which features dialogue trees across a massive document corpus. This corpus includes approximately 5 million Wikipedia articles (KILT), 11.9 million MS MARCO V2 documents, and 728,000 news articles from the TREC Washington Post (V4) collection.
Results and Analysis
The performance was evaluated using Mean Average Precision (MAP) and Normalized Discounted Cumulative Gain (nDCG@20) under both "Lenient" and "Strict" relevance settings.
The initial retrieval results (shown above) demonstrate the baseline effectiveness of each strategy. Notably, CNR-run2, which utilizes manual rewrites, consistently outperformed the other runs, highlighting the gap between current automated context resolution and human-level understanding. The table also includes track-wide averages (Min, Median, Max) for comparison.
The application of the BERT re-ranker provided a significant performance boost across all metrics. For example, CNR-run2's nDCG@20 improved from 0.1832 to 0.2200. Interestingly, CNR-run3 (using the first sentence of the previous response) outperformed CNR-run1 (using automatic rewrites). This suggests that the system's own previous answer can be a more reliable context source than a potentially noisy automated rewrite of a previous query.
Conclusion
The CNR team's experiments demonstrate that while manual context remains the gold standard, leveraging the system's previous responses is a promising avenue for automatic query enrichment. Future work will focus on more sophisticated dependency modeling to better capture the specific relationships between dialogue turns and user intent.
Summary updated