Abstract
In conversational question answering (CQA), the task of question rewriting (QR) in context aims to rewrite a context-dependent question into an equivalent self-contained question that gives the same answer. In this paper, we are interested in the robustness of a QR system to questions varying in rewriting hardness or difficulty. Since there is a lack of questions classified based on their rewriting hardness, we first propose a heuristic method to automatically classify questions into subsets of varying hardness, by measuring the discrepancy between a question and its rewrite. To find out what makes questions hard or easy for rewriting, we then conduct a human evaluation to annotate the rewriting hardness of questions. Finally, to enhance the robustness of QR systems to questions of varying hardness, we propose a novel learning framework for QR that first trains a QR model independently on each subset of questions of a certain level of hardness, then combines these QR models as one joint model for inference. Experimental results on two datasets show that our framework improves the overall performance compared to the baselines.
Introduction
Conversational Question Answering (CQA) systems must navigate the nuances of natural dialogue, where users often rely on context and pronouns. The task of Question Rewriting (QR) is critical here: it involves transforming a context-dependent query (e.g., "When was he elected?") into a standalone version (e.g., "When was Benigno Aquino III elected?") that can be processed by standard retrieval systems.
However, current QR research often overlooks the fact that not all questions are equally difficult to rewrite. While simple pronoun replacement is straightforward, complex queries requiring the expansion of "Wh*" phrases or the integration of distant conversational context present a much greater challenge. This paper addresses the robustness of QR systems across these varying levels of "rewriting hardness."
Methodology
To address the lack of labeled data for difficulty, the researchers proposed a heuristic based on the BLEU score between an original question and its human-authored rewrite. A low BLEU score indicates a significant transformation, categorizing the question as "hard," while a high score suggests an "easy" rewrite.
The proposed framework utilizes a shared-private architecture built upon the BART language model. Small, trainable neural network layers called adapters are inserted into the model. Separate adapters are trained for "hard," "medium," and "easy" subsets of the data. To integrate these specialized components during inference, two methods were developed:
- Sequence-level Adapter Fusion (SLAF): Uses a classifier to dynamically weight the outputs of the specialized adapters.
- Sequence-level Adapter Distillation (SLAD): Distills the collective knowledge of the specialized adapters into a single, efficient student model.
Results and Analysis
The study highlights a stark "difficulty gap" in QR performance. On the CANARD dataset, baseline models achieve high scores on easy questions but struggle significantly as difficulty increases. Human evaluation confirmed that "easy" questions typically involve simple coreference resolution, whereas "hard" questions require more complex generative capabilities to complete sentences or expand "Wh*" questions.
The specialized framework consistently outperformed standard BART baselines. On CANARD, the SLAD model reached a mean BLEU of 53.35, compared to 52.77 for the baseline. Notably, models trained specifically on the "hard" subset (P-hard) showed the most significant improvements in that category, suggesting that specialization is key to handling complex context.
Conclusion
This research demonstrates that categorizing questions by rewriting hardness is a viable path toward more robust CQA systems. By employing specialized adapters and fusion/distillation techniques, the proposed framework mitigates the performance drop-off on difficult queries. Future work will focus on improving question classification, as the system's current bottleneck is accurately predicting a question's difficulty level without prior knowledge of the target rewrite.
Summary updated