Issues for the Evaluation of Ambient Displays

Xiaobin Shen, Andrew Vande Moere, Peter Eades, Seok-Hee HongInt. J. Ambient Comput. Intell 2009 · DOI

TL;DR

This paper explores the challenges of evaluating "ambient displays" – systems designed to convey information peripherally without demanding full attention. Through two case studies, "MoneyColor" and "Fisherman," the authors investigate the trade-offs between display distraction and comprehension, and propose a quantitative measure for user "interest" in ambient displays, highlighting the need for stable interest levels before conducting evaluations.

What Problem Does This Paper Address?

Ambient displays are a fascinating subfield of information visualization aiming to present information subtly, often in the periphery of a user's attention, drawing inspiration from the ubiquitous computing paradigm. Unlike traditional information visualizations that demand focused user interaction, ambient displays are designed to be unobtrusive, allowing users to engage in other tasks. However, the evaluation of these displays presents a significant challenge. Traditional evaluation methods, often intrusive and requiring full user attention, are not well-suited for systems designed to operate in the background. This paper addresses the gap in robust and appropriate evaluation methodologies for ambient displays, exploring how to measure their effectiveness and user engagement without disrupting the very ambient nature they aim to achieve.

Methods

The paper presents two distinct case studies employing different evaluation methodologies:

  • Intrusive Evaluation (MoneyColor): This study used a laboratory setting with two monitors. Subjects performed a primary task on one monitor while the ambient display (MoneyColor, showing stock market data) was on a secondary, peripheral monitor. The evaluation focused on measuring the relationship between "distraction" and "comprehension." Key parameters analyzed were:
  • Mean Comprehension Rate (MCR): Measured by questionnaire, assessing recall of information from the peripheral display.
  • Mean Self-Interruption (MSI): Counted focus shifts to the peripheral screen initiated by the user.
  • Mean Display-Distraction (MDD): Counted focus shifts caused by the display itself. A within-subject experimental design was used.

  • Non-Intrusive Evaluation (Fisherman): This study took place in a real-life, semi-public environment. The Fisherman display, a visualization of stock price movements in a Chinese watercolor style, was placed in a frame with an IR sensor and a video camera. The evaluation aimed to quantitatively measure "user interest" without direct user interruption. Methods included:

  • Face Detection and Recognition: Used to count individuals passing by and those who faced the display for a significant duration.
  • IR Sensor: Assisted in counting passers-by.
  • Questionnaire/Interviews: Used to gather additional data on comprehension. The study lasted five months, collecting data from all passers-by.

Datasets

Not discussed in the paper. The paper focuses on the design and evaluation of specific ambient display systems rather than using pre-existing datasets.

Key Insights and Ideas

  • Distraction vs. Comprehension Trade-off: The MoneyColor study revealed a direct proportional relationship between display-distraction (MDD) and comprehension (MCR). This suggests that a more visually engaging or attention-grabbing ambient display might lead to better understanding of the information it presents, up to a point.
  • Self-Interruption is Different: Interestingly, there was no clear relationship found between self-interruption (MSI) and comprehension. This highlights that while the display itself can cause distraction, a user's decision to look at it is influenced by different factors (personality, curiosity) and doesn't directly correlate with their ability to understand the information.
  • Defining and Measuring "User Interest": The Fisherman study proposed a novel, quantitative measure for user interest: the ratio of total subjects looking at the display (TSL) to the total subjects passing by (TSP). This provides a concrete way to assess how engaging an ambient display is in a real-world context.
  • The Importance of "Evaluation Time": The authors argue that ambient display evaluations should be delayed until user interest stabilizes. This stable interest indicates the display has become integrated into the environment, meeting the goal of ambient displays to inform without being overly distracting or burdensome.
  • Intrusive vs. Non-Intrusive Trade-offs: The paper clearly delineates the strengths and weaknesses of each approach. Intrusive evaluations offer better control and quantitative measurement but may lack ecological validity. Non-intrusive evaluations provide real-world insights but face significant ethical and technological challenges.

Key Results

  • MoneyColor: Found that Mean Comprehension Rate (MCR) is indirectly proportional to Mean Display-Distraction (MDD). No significant relationship was found between MCR and Mean Self-Interruption (MSI).
  • Fisherman: User interest, measured as TSL/TSP, initially peaked and then stabilized over the 20-week experiment. This stabilization is attributed to the display becoming a familiar part of the environment. Face recognition data suggested some participants revisited the display, indicating sustained interest.

Limitations and Future Work

  • Ethical Concerns in Non-Intrusive Evaluation: The paper acknowledges the significant ethical challenges in non-intrusive evaluations, particularly regarding privacy in public spaces. While they implemented measures like recording only face counts, not actual video, these issues remain a major constraint.
  • Technological Limitations: The accuracy of face detection and recognition in non-laboratory settings was noted as a limitation (around 30% error rate for face detection), impacting the reliability of some data. The authors suggest that many potentially useful non-intrusive techniques are still under development.
  • Defining the Boundary: The authors note that the distinction between intrusive and non-intrusive evaluation is not a strict dichotomy but rather a spectrum. Experiments can inadvertently become intrusive.
  • Future Work: The authors plan to conduct more experiments to further validate their conclusions and explore more robust evaluation methodologies for ambient displays, particularly addressing the technological and ethical hurdles of non-intrusive approaches. They also suggest that the evaluation timeline for ambient displays should be carefully considered, waiting for user interest to stabilize.

Summary updated

BibTeX
@article{shen2009issues,
  author = {Xiaobin Shen and Andrew Vande Moere and Peter Eades and Seok-Hee Hong},
  title = {{Issues for the Evaluation of Ambient Displays}},
  journal = {Int. J. Ambient Comput. Intell.},
  year = {2009},
  doi = {10.4018/jaci.2009040108},
}
Figure