Adapting visualizations and interfaces to the user

Francesco Chiossi, Johannes Zagermann, Jakob Karolus, Nils Rodrigues, Priscilla Balestrucci, Daniel Weiskopf, Benedikt V. Ehinger, Tiare M. Feuchtner, Harald Reiterer, Lewis L. Chuang, Marc O. Ernst, Andreas Bulling, Sven Mayer, Albrecht Schmidt

Introduction

Traditional Human-Computer Interaction (HCI) is fundamentally reactive: the computer waits for an explicit command—a click, a keystroke, or a voice prompt—and then responds. This creates a significant bottleneck in high-complexity environments. If a user is overwhelmed by a dense visualization, fatigued by poorly placed 3D menus, or struggling with a language barrier, a static interface remains passive. This paper explores the transition toward "biocybernetic" systems that leverage implicit interaction to proactively adapt to a user’s physiological and cognitive state.

The Biocybernetic Framework

The authors categorize the development of adaptive systems into three core pillars:

  1. Sensing (Implicit Input): Utilizing psychophysiological measures such as EEG to estimate cognitive workload, eye tracking to infer interest or language proficiency, and Electrodermal Activity (EDA) to measure stress.
  2. Adaptation Elements: Determining what changes in the interface, including content (simplifying text), presentation (repositioning notifications), and interaction (switching control schemes based on fatigue).
  3. Biocybernetic Loops: Implementing first-order loops for immediate action and second-order loops that detect the result of an adaptation to refine future system responses.

Key Applications and Insights

Language-Aware Interfaces

Gaze patterns serve as a unique fingerprint for comprehension. Research indicates that a system can detect within seconds whether a user understands the language on screen by analyzing fixation and blink durations. This allows for real-time content adaptation, such as providing translations or simplified text when a user is struggling.

Gaze patterns as a fingerprint for language comprehension, illustrating how fixation points differ based on language proficiency.

XRgonomics and VR Adaptation

In Mixed Reality (MR) and Virtual Reality (VR), physical fatigue is a major usability hurdle. The authors introduce "XRgonomics," using metrics like "Consumed Endurance" to automatically place 3D UI elements in the most ergonomic positions relative to the user's current posture. Additionally, systems can manage cognitive load by sensing rising stress via EDA and dynamically removing "visual noise," such as background characters, to help the user focus on primary tasks.

A user in a VR environment where biocybernetic loops can optimize UI placement and manage cognitive load through physiological sensing.

The "Midas-Touch" and Sensor Fusion

A persistent challenge in gaze-based interaction is the "Midas-Touch" problem: distinguishing between a user looking at an object out of interest versus an intent to interact. The paper highlights how sensor fusion—combining EEG with eye tracking—can resolve this by estimating if a fixation was purposeful, thereby reducing accidental triggers.

Challenges and Future Directions

While promising, the path to fully adaptive systems faces several "grand challenges":

  • Construct Validity: There is rarely a one-to-one mapping between a physiological spike and a specific cognitive state, making accurate interpretation difficult.
  • Privacy and Agency: Physiological data is highly personal. Because users cannot easily control their heart rate or skin conductance, these systems create a power asymmetry that requires careful ethical consideration.
  • Mutual Accountability: The community continues to debate the balance of control—whether the system should adapt autonomously or if the user should always maintain the final say in interface changes.

Conclusion

By "closing the loop" between the user and the system, biocybernetic interfaces offer a path toward more supportive and efficient computing. Moving beyond reactive commands allows technology to meet users where they are, optimizing performance and reducing cognitive strain in increasingly complex digital landscapes.

Summary updated

Abstract

Adaptive visualization and interfaces pervade our everyday tasks to improve interaction from the point of view of user performance and experience. This approach allows using several user inputs, whether physiological, behavioral, qualitative, or multimodal combinations, to enhance the interaction. Due to the multitude of approaches, we outline the current research trends of inputs used to adapt visualizations and user interfaces. Moreover, we discuss methodological approaches used in mixed reality, physiological computing, visual analytics, and proficiency-aware systems. With this work, we provide an overview of the current research in adaptive systems.
BibTeX
@article{chiossi2022adapting,
  author = {Francesco Chiossi and Johannes Zagermann and Jakob Karolus and Nils Rodrigues and Priscilla Balestrucci and Daniel Weiskopf and Benedikt V. Ehinger and Tiare M. Feuchtner and Harald Reiterer and Lewis L. Chuang and Marc O. Ernst and Andreas Bulling and Sven Mayer and Albrecht Schmidt},
  title = {{Adapting visualizations and interfaces to the user}},
  journal = {It Inf. Technol},
  year = {2022},
  doi = {10.1515/itit-2022-0035},
}
Figure