Abstract
Current research on Language User Interfaces (LUIs) is increasingly confined to a "benchmark trap," where models achieve high scores on artificial datasets but fail to provide value in real-world settings. This paper critiques the prioritization of data quantity over naturalness and proposes a return to "ecological validity." By aligning research procedures with actual human-task interactions, we can develop interfaces that solve genuine problems.
The Ecological Validity Crisis
Despite significant advances in deep learning, systems like Siri or natural language database tools often feel limited. The authors argue this stems from a lack of ecological validity: the extent to which research findings generalize to real-world scenarios. In the pursuit of large-scale training data, researchers have created benchmarks that are easy to scale but do not reflect how humans naturally use language to accomplish tasks. Consequently, progress on these benchmarks rarely translates to improved user experiences.
A Framework for Valid Research
To address these shortcomings, the paper proposes an Ideal Ecologically Valid Research Procedure:
- Identify the User and Task: Focus on a specific population and a concrete task where an LUI provides clear utility.
- Wizard-of-Oz (WoZ) Collection: Use a human "wizard" to simulate the system, capturing naturalistic user behavior.
- Model Training: Train models on this high-quality, representative data.
- Human-in-the-Loop Evaluation: Measure success through user satisfaction and task completion.
Categorizing Benchmark Deviations
The authors identify five primary ways current benchmarks deviate from ecological validity:
- Synthetic Language: Using templates that lack linguistic variety (e.g., CLEVR).
- Artificial Tasks: Focusing on games rather than productive work (e.g., CerealBar).
- Non-Prospective Users: Collecting data from crowd workers who do not share the goals of the intended users (e.g., the "Stump the Robot" fallacy in VQA).
- Scripts and Priming: Guiding users with rigid instructions that limit natural expression.
- Single-turn Interfaces: Ignoring the context-dependent nature of real conversation.
Insights from Naturalistic Data
Real-world interaction is messy and iterative. For example, in task-oriented dialogues, users frequently rely on linguistic shortcuts like anaphora (e.g., "it," "them") and often change their minds mid-stream.
The paper notes that scripting in datasets like MultiWOZ "primes" users to use specific vocabulary, making models brittle. Furthermore, while 36% of real-world QA questions rely on previous context, many benchmarks remain strictly single-turn.
Conclusion and Future Directions
Collecting ecologically valid data is expensive and difficult to scale compared to traditional crowdsourcing. However, the authors suggest leveraging existing environments (like Minecraft) for data collection and shifting the focus toward valid evaluation metrics. The core recommendation is to stop pursuing incremental gains on flawed benchmarks and instead prioritize the development of systems that genuinely assist human users in their tasks.
Summary updated