The Disconnect Between Subjective Preference and Objective Performance

145 concepts96 papers1 proposal

In the design of intelligent systems, a common assumption prevails: if a user prefers a specific interface, finds an explanation helpful, or rates a system as highly intelligent, that system is likely improving their performance. However, a growing body of research across natural language generation (NLG), explainable AI (XAI), and human-computer interaction reveals a persistent and often stark preference-performance gap.

This phenomenon occurs when subjective measures of user satisfaction, trust, and perceived utility decouple from objective metrics like task accuracy, decision speed, and learning outcomes. Understanding why this gap exists is critical for building AI that actually assists humans rather than merely pleasing them.

The Illusion of Understanding

One of the most frequent sites of this disconnect is in Explainable AI. We often assume that a "good" explanation is one that a user finds intuitive or satisfying. Yet, Hase and Bansal demonstrated that users’ subjective ratings of an explanation’s quality are unreliable predictors of simulatability—the objective ability of a user to predict how the model will behave on new data. An explanation can create an "illusion of explainability," where the user feels they understand the system better even when their mental model remains flawed.

This discrepancy is particularly visible in natural language interfaces. Forrest et al. found that while users subjectively preferred narrative, text-based explanations over raw data tables, their objective comprehension of the underlying facts did not improve. Similarly, Balepur et al. found that in educational settings, what students say they like in a mnemonic device often differs from what actually helps them retain information long-term.

The Over-reliance Trap

When systems are designed to be highly persuasive or "helpful," they can inadvertently trigger over-reliance. Bansal et al. observed that adding explanations to AI recommendations often failed to improve team accuracy because humans began following the AI even when it was incorrect. The explanation served to build trust, but not the calibrated trust necessary for error detection.

This is especially dangerous in high-stakes domains like medicine. Kayser found that healthcare practitioners strongly preferred natural language justifications for AI advice, yet these very explanations led to higher error rates when the AI was wrong. In contrast, less "preferred" visual cues were sometimes more effective at signaling model failures. As Gaube et al. noted, there is often a gap between a professional's "algorithmic aversion" (stated distrust) and their actual behavioral reliance on the system's suggestions.

Cognitive Load and the Cost of Helpfulness

Why does a "better" interface sometimes lead to worse performance? The answer often lies in cognitive overhead. Paleja et al. showed that while detailed explanations can increase a user's perception of a robot's intelligence, they can simultaneously hinder expert performance by introducing a "preoccupation cost." The effort required to process the explanation outweighs the utility of the information provided.

This friction is also evident in software development. Mozannar et al. found that developers' subjective preferences for coding assistants were often biased by factors like latency or conversational quality, rather than whether the tool actually saved them time or reduced bugs. In some cases, users reported that AI-generated highlights were helpful even when those highlights statistically slowed them down and decreased their accuracy (Kim et al.).

Expert Dismissal and Sensory Gaps

Conversely, the gap can work in the opposite direction: a system may be objectively superior but subjectively rejected. Hershcovich et al. found that professional journalists sometimes dismissed AI-generated rationales as "random" or "biased" even when those rationales measurably improved their ability to detect fake news.

In physical systems, like automated driving, the gap is often sensory. Lei et al. argued that engineering benchmarks for parking assist systems often focus on spatial accuracy, while users judge the system based on operational smoothness and perceived safety. A system can be mathematically perfect at a task while failing to meet the user's psychological expectations for how a "good" driver should behave.

Moving Beyond Satisfaction Metrics

The recurring evidence of this gap suggests that user preference is a poor proxy for system effectiveness. As Reiter et al. famously demonstrated in a clinical trial of an NLG system, users may love a personalized, high-quality output that nonetheless fails to achieve its real-world goal (e.g., smoking cessation).

To bridge this gap, researchers are moving toward dual-metric evaluations that explicitly contrast:

  1. Subjective Utility: How much the user likes, trusts, or values the system.
  2. Objective Performance: Measurable gains in accuracy, speed, or behavioral change.

By measuring both, as suggested by Nagendran et al. through eye-tracking and behavioral analysis, we can identify when an interface is merely "charming" the user and when it is truly empowering them. Relying on what users say they want may lead to systems that are pleasant to use but functionally inert; only by measuring what users do can we ensure that AI assistance translates into actual performance.

Key Foundational Work

Using a Randomised Controlled Clinical Trial to Evaluate an NLG System (2001) — contributed 2 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 86 papers
  2. Version 1 · 2026-10-04 · 94 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_preference_performance_gap_3c5caf39,
  title = {preference-performance gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/preference-performance-gap/rdr_3c5caf3941ec4ba69aec97a447125147},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_preference_performance_gap_3c5caf39_v2,
  title = {preference-performance gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/preference-performance-gap/rdr_3c5caf3941ec4ba69aec97a447125147/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake