Introduction: The "Safe Response" Trap
Conversational AI often falls into a "Safe Response" trap, defaulting to generic, uninformative phrases like "I don't know" or "I'm not sure" instead of providing helpful answers. While these responses are grammatically correct, they fail the fundamental goal of communication. This research frames the issue using Gricean Maxims, specifically the Maxim of Quantity, which dictates that a speaker should be as informative as required for the current exchange.
When a chatbot defaults to a generic response despite having a more specific, relevant answer available in its training data, it commits a "Quantity violation." This paper introduces a diagnostic framework to measure and mitigate this specific failure.
Methodology: Relative Utterance Quantity (RUQ)
To diagnose why models prefer safe responses, the authors propose Relative Utterance Quantity (RUQ). This metric compares the model's internal probability score for a generic response against its score for the actual ground-truth reference from the training data.
The researchers employed two primary diagnostic tools:
- RUQ Plots: These visualize the average model score for each token across a sentence, revealing where a model "loses interest" in a specific reference response compared to a generic one.
- RUQ Score: A summary statistic representing the percentage of instances where the model assigns a higher probability to the ground-truth reference than to a generic "I don't know" response.
The study compared "BASE" models (standard settings) against "BEST" models, which were optimized through an exhaustive hyperparameter sweep involving label smoothing, dropout, and subword vocabulary adjustments.
Datasets
Experiments were conducted on two versions of the same source:
- DailyDialog: A high-quality dataset of ~80,000 everyday conversational turns.
- Entropy-Filtered (EF) DailyDialog: A variant where highly generic (high-entropy) utterances were removed to test if data cleaning alone could solve the problem.
Key Insights
The study revealed that baseline models often exhibit a "Preference for Ignorance," prioritizing generic responses over the specific information they were explicitly trained on. Interestingly, while filtering training data (the EF method) provided minor improvements, hyperparameter tuning proved far more effective.
Techniques like label smoothing—which prevents overconfidence—and optimizing subword vocabulary size were critical. These engineering choices significantly increased the model's "willingness" to provide informative content. RUQ serves as a vital intrinsic check, revealing whether a model has truly learned to value information over safety.
Key Results
The transition from baseline configurations to optimized hyperparameters yielded a massive quantitative shift in model behavior.
As shown in the results, the RUQ score for DailyDialog models jumped from a mere 28.5% in the BASE model to 95.3% in the BEST model. This indicates that the optimized model prefers the informative reference nearly every time. This improvement was mirrored in standard metrics like METEOR (rising from 12.7 to 17.8) and confirmed by human evaluations, which rated the "BEST" models as significantly more interesting and coherent.
Conclusion and Future Work
RUQ provides a targeted lens for identifying uninformative model behavior. While it effectively captures violations of the Maxim of Quantity (under-informativeness), it does not yet address the opposite problem of "Too Much Information" (TMI). Future research may expand this framework to other Gricean Maxims, such as Relation (relevance) and Quality (truthfulness), to create a more holistic evaluation of conversational AI.
Summary updated