Abstract
User preferences for automated assistance often vary widely, depending on the situation, and quality or presentation of help. Developing effective models to learn individual preferences online requires domain models that associate observations of user behavior with their utility functions, which in turn can be constructed using utility elicitation techniques. However, most elicitation methods ask for users' predicted utilities based on hypothetical scenarios rather than more realistic experienced utilities. This is especially true in interface customization, where users are asked to assess novel interface designs. We propose experiential utility elicitation methods for customization and compare these to predictive methods. As experienced utilities have been argued to better reflect true preferences in behavioral decision making, the purpose here is to investigate accurate and efficient procedures that are suitable for software domains. Unlike conventional elicitation, our results indicate that an experiential approach helps people understand stochastic outcomes, as well as better appreciate the sequential utility of intelligent assistance.
Introduction
Modern intelligent software—ranging from adaptive toolbars to predictive text systems—relies on understanding a user's "utility function." This function quantifies the trade-off between the benefits of automated assistance and the costs of interruption or cognitive load. Traditionally, developers have relied on conceptual elicitation, where users are presented with static descriptions or screenshots of an interface and asked to predict their preference on a numerical scale.
However, humans are notoriously poor at forecasting their own preferences for novel experiences. A user might conceptually dislike a pop-up menu but find it indispensable during a high-pressure task. This paper investigates experiential utility elicitation, a method where users perform actual tasks to "feel" the utility of an interface before providing a rating.
Experimental Methodology
To compare elicitation methods, the researchers developed a controlled environment using a custom plugin for PowerPoint 2003. Participants were tasked with a repetitive highlighting exercise, applying specific font attributes to text strings.
The study involved 30 participants divided into four experimental groups:
- Conceptual: Users rated interfaces based on descriptions and screenshots.
- Experiential: Users completed 20 mini-tasks across different interface configurations before rating them.
- Primed: A training session followed by conceptual questions.
- Primed+: A hybrid approach combining training with a small number of experiential "calibration" tasks.
The interfaces varied in complexity (number of icons) and quality (the accuracy of the automated font settings).
Results: The Underestimation Bias
The core finding of the study is that users consistently underestimate the value of intelligent assistance when they cannot experience it firsthand. In the conceptual group, utility ratings for helpful features were significantly lower than those provided by the experiential group.
Statistical analysis using Hotelling’s $T^2$ statistic confirmed that the mean utilities elicited via experience were significantly different ($p < 0.01$) from those elicited conceptually. As shown in the t-score analysis below, the experiential group frequently reported higher utility across various configurations of icon count (N), likelihood of success (L), and quality (Q).
This discrepancy suggests that "sequential utility"—the cumulative benefit of saved clicks and reduced cognitive load over time—is difficult to conceptualize without direct experience. Furthermore, users struggled to accurately predict their reaction to stochastic (probabilistic) outcomes. Experiencing a system that is 80% accurate provided a much more realistic basis for preference than simply being told the success rate.
The "Sweet Spot": Primed+ Elicitation
While experiential elicitation provides more accurate data, it is time-intensive. The experiential group required approximately 2 hours per user, compared to just 30 minutes for the conceptual group.
The researchers found a middle ground in the Primed+ method. By combining a training session with just five experiential tasks, the system could "calibrate" the user's expectations. This method took roughly 60 minutes—half the time of the full experiential version—yet produced utility curves that closely mirrored the more accurate experiential data.
Conclusion
Experiential elicitation reveals that users value intelligent assistance more than they realize. While direct experience is costly in terms of time and potential user fatigue, hybrid methods like Primed+ offer a viable path for training adaptive interfaces. Future research will focus on scaling these methods to larger state spaces using parametric utility models, allowing systems to predict preferences across a wider range of features with minimal user queries.
Summary updated