Evaluating Variable-Length Multiple-Option Lists in Chatbots and Mobile Search

Pepa Atanasova, Georgi Karadzhov, Yasen Kiprov, Preslav Nakov, Fabrizio Sebastiani

Abstract

In recent years, the proliferation of smart mobile devices has lead to the gradual integration of search functionality within mobile platforms. This has created an incentive to move away from the "ten blue links'' metaphor, as mobile users are less likely to click on them, expecting to get the answer directly from the snippets. In turn, this has revived the interest in Question Answering. Then, along came chatbots, conversational systems, and messaging platforms, where the user needs could be better served with the system asking follow-up questions in order to better understand the user's intent. While typically a user would expect a single response at any utterance, a system could also return multiple options for the user to select from, based on different system understandings of the user's intent. However, this possibility should not be overused, as this practice could confuse and/or annoy the user. How to produce good variable-length lists, given the conflicting objectives of staying short while maximizing the likelihood of having a correct answer included in the list, is an underexplored problem. It is also unclear how to evaluate a system that tries to do that. Here we aim to bridge this gap. In particular, we define some necessary and some optional properties that an evaluation measure fit for this purpose should have. We further show that existing evaluation measures from the IR tradition are not entirely suitable for this setup, and we propose novel evaluation measures that address it satisfactorily.

Introduction

Modern user experiences, particularly on mobile devices and within conversational interfaces, have moved beyond the traditional "ten blue links" paradigm. When a chatbot is uncertain about a user's intent, it often presents a list of possible options to clarify the request. This creates a unique challenge: the system must balance Recall (ensuring the correct answer is present) with Conciseness (minimizing the list length to avoid clutter and user annoyance).

Figure 2: Examples of chatbot interactions where the system provides multiple options to clarify ambiguous user intents.

As shown in Figure 2, a chatbot might offer options to replace a broken card or report a stolen one when a user says their card is "toast." While helpful, over-returning results on small screens is detrimental. Traditional Information Retrieval (IR) metrics often fail to penalize the "clutter" of extra, irrelevant options at the end of these short lists.

Evaluation Desiderata

The researchers defined three fundamental properties that any effective metric for variable-length suggestion lists must satisfy:

  1. Correctness: A list containing the correct answer is always superior to one that does not.
  2. Confidence: If two lists both contain the correct answer, the shorter list is better (less clutter).
  3. Priority: If two lists have the same length and both contain the correct answer, the one that ranks the correct answer higher is better.

Proposed Metrics: LAR and OLAR

Finding that standard metrics like F1 and Average Precision (AP) failed to satisfy the "Confidence" property, the authors proposed two new measures:

  • Length-Aware Recall (LAR): Designed for unranked sets, LAR calculates the mean of the recall and the reciprocal of the list length ($1/|r_n|$). This explicitly rewards brevity.
  • Ordered Length-Aware Recall (OLAR): An extension for ranked lists, OLAR incorporates a term for the position of the correct answer, ensuring that "Priority" is respected without allowing it to override the need for conciseness.

Experimental Results

To validate these metrics, the authors used a synthetic "gold ranking" benchmark. They generated all possible combinations of response lists up to five items (e.g., 'c' for correct, 'w' for wrong) and manually ranked them according to the three desiderata.

Figure 3: Comparison of various evaluation metrics against a gold standard ranking of result lists. LAR and OLAR are the only metrics to achieve perfect correlation (1.0) and satisfy all three desiderata.

As detailed in Figure 3, popular metrics such as nDCG, MRR, and AP failed the "Confidence" test, as they could not distinguish between a perfect single-answer response ('c') and a response containing the correct answer followed by noise ('cw'). In contrast, both LAR and OLAR achieved a perfect correlation of 1.0 (Kendall’s Tau and Spearman) with the gold ranking, proving their robustness in evaluating conversational suggestion lists.

Conclusion

This research highlights the limitations of traditional IR metrics in the context of modern, screen-constrained conversational interfaces. By introducing LAR and OLAR, the authors provide a mathematical framework that aligns evaluation with actual user preferences for accuracy and brevity. Future work will explore multi-intent scenarios where a user may have more than one "correct" goal in a single utterance.

Summary updated

Abstract

In recent years, the proliferation of smart mobile devices has lead to the gradual integration of search functionality within mobile platforms. This has created an incentive to move away from the "ten blue links'' metaphor, as mobile users are less likely to click on them, expecting to get the answer directly from the snippets. In turn, this has revived the interest in Question Answering. Then, along came chatbots, conversational systems, and messaging platforms, where the user needs could be better served with the system asking follow-up questions in order to better understand the user's intent. While typically a user would expect a single response at any utterance, a system could also return multiple options for the user to select from, based on different system understandings of the user's intent. However, this possibility should not be overused, as this practice could confuse and/or annoy the user. How to produce good variable-length lists, given the conflicting objectives of staying short while maximizing the likelihood of having a correct answer included in the list, is an underexplored problem. It is also unclear how to evaluate a system that tries to do that. Here we aim to bridge this gap. In particular, we define some necessary and some optional properties that an evaluation measure fit for this purpose should have. We further show that existing evaluation measures from the IR tradition are not entirely suitable for this setup, and we propose novel evaluation measures that address it satisfactorily.
BibTeX
@inproceedings{atanasova2019evaluating,
  author = {Pepa Atanasova and Georgi Karadzhov and Yasen Kiprov and Preslav Nakov and Fabrizio Sebastiani},
  title = {{Evaluating Variable-Length Multiple-Option Lists in Chatbots and Mobile Search}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2019},
  url = {https://openreview.net/forum?id=tdrF-Op0nS},
}
Figure