Towards Ecologically Valid Research on Language User Interfaces

Harm de Vries, Dzmitry Bahdanau, Christopher D. ManningarXiv 2020 · arXiv

Abstract

Current research on Language User Interfaces (LUIs) is increasingly confined to a "benchmark trap," where models achieve high scores on artificial datasets but fail to provide value in real-world settings. This paper critiques the prioritization of data quantity over naturalness and proposes a return to "ecological validity." By aligning research procedures with actual human-task interactions, we can develop interfaces that solve genuine problems.

The Ecological Validity Crisis

Despite significant advances in deep learning, systems like Siri or natural language database tools often feel limited. The authors argue this stems from a lack of ecological validity: the extent to which research findings generalize to real-world scenarios. In the pursuit of large-scale training data, researchers have created benchmarks that are easy to scale but do not reflect how humans naturally use language to accomplish tasks. Consequently, progress on these benchmarks rarely translates to improved user experiences.

A Framework for Valid Research

To address these shortcomings, the paper proposes an Ideal Ecologically Valid Research Procedure:

  1. Identify the User and Task: Focus on a specific population and a concrete task where an LUI provides clear utility.
  2. Wizard-of-Oz (WoZ) Collection: Use a human "wizard" to simulate the system, capturing naturalistic user behavior.
  3. Model Training: Train models on this high-quality, representative data.
  4. Human-in-the-Loop Evaluation: Measure success through user satisfaction and task completion.
The Wizard-of-Oz (WoZ) framework for capturing naturalistic interaction data.

Categorizing Benchmark Deviations

The authors identify five primary ways current benchmarks deviate from ecological validity:

  • Synthetic Language: Using templates that lack linguistic variety (e.g., CLEVR).
  • Artificial Tasks: Focusing on games rather than productive work (e.g., CerealBar).
  • Non-Prospective Users: Collecting data from crowd workers who do not share the goals of the intended users (e.g., the "Stump the Robot" fallacy in VQA).
  • Scripts and Priming: Guiding users with rigid instructions that limit natural expression.
  • Single-turn Interfaces: Ignoring the context-dependent nature of real conversation.

Insights from Naturalistic Data

Real-world interaction is messy and iterative. For example, in task-oriented dialogues, users frequently rely on linguistic shortcuts like anaphora (e.g., "it," "them") and often change their minds mid-stream.

An example of a multi-turn dialogue where the user iteratively refines their request, a common pattern in real-world tasks.

The paper notes that scripting in datasets like MultiWOZ "primes" users to use specific vocabulary, making models brittle. Furthermore, while 36% of real-world QA questions rely on previous context, many benchmarks remain strictly single-turn.

Conclusion and Future Directions

Collecting ecologically valid data is expensive and difficult to scale compared to traditional crowdsourcing. However, the authors suggest leveraging existing environments (like Minecraft) for data collection and shifting the focus toward valid evaluation metrics. The core recommendation is to stop pursuing incremental gains on flawed benchmarks and instead prioritize the development of systems that genuinely assist human users in their tasks.

Summary updated

Abstract

Language User Interfaces (LUIs) could improve human-machine interaction for a wide variety of tasks, such as playing music, getting insights from databases, or instructing domestic robots. In contrast to traditional hand-crafted approaches, recent work attempts to build LUIs in a data-driven way using modern deep learning methods. To satisfy the data needs of such learning algorithms, researchers have constructed benchmarks that emphasize the quantity of collected data at the cost of its naturalness and relevance to real-world LUI use cases. As a consequence, research findings on such benchmarks might not be relevant for developing practical LUIs. The goal of this paper is to bootstrap the discussion around this issue, which we refer to as the benchmarks' low ecological validity. To this end, we describe what we deem an ideal methodology for machine learning research on LUIs and categorize five common ways in which recent benchmarks deviate from it. We give concrete examples of the five kinds of deviations and their consequences. Lastly, we offer a number of recommendations as to how to increase the ecological validity of machine learning research on LUIs.
BibTeX
@misc{vries2020towards,
  author = {Harm de Vries and Dzmitry Bahdanau and Christopher D. Manning},
  title = {{Towards Ecologically Valid Research on Language User Interfaces}},
  year = {2020},
  eprint = {2007.14435},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.14435},
}
Figure