Continuous Mental Effort Evaluation During 3D Object Manipulation Tasks Based on Brain and Physiological Signals

Dennis Wobrock, Jérémy Frey, Delphine Graeff, Jean-Baptiste de la Rivière, Julien Castet, Fabien Lotte

Abstract

Designing 3D User Interfaces (UI) requires adequate evaluation tools to ensure good usability and user experience. While many evaluation tools are already available and widely used, existing approaches generally cannot provide continuous and objective measures of usability qualities during interaction without interrupting the user. In this paper, we propose to use brain (with ElectroEncephaloGraphy) and physiological (ElectroCardioGraphy, Galvanic Skin Response) signals to continuously assess the mental effort made by the user to perform 3D object manipulation tasks. We first show how this mental effort (a.k.a., mental workload) can be estimated from such signals, and then measure it on 8 participants during an actual 3D object manipulation task with an input device known as the CubTile. Our results suggest that monitoring workload enables us to continuously assess the 3DUI and/or interaction technique ease-of-use. Overall, this suggests that this new measure could become a useful addition to the repertoire of available evaluation tools, enabling a finer grain assessment of the ergonomic qualities of a given 3D user interface.

Introduction

Designing effective 3D User Interfaces (UIs) is a complex challenge. 3D interaction is inherently cognitively demanding, often requiring mental rotations and sophisticated spatial reasoning. Traditionally, designers have relied on "think-aloud" protocols, behavioral metrics like reaction time, or post-hoc questionnaires such as the NASA-TLX to evaluate these interfaces.

However, these traditional methods suffer from several critical flaws. Subjective questionnaires rely on user memory and can be biased by social pressure. "Think-aloud" protocols are disruptive, potentially altering the very mental workload they aim to measure. Furthermore, most existing tools lack granularity; they provide a single global score for a task but cannot pinpoint exactly when a user struggled during a multi-minute interaction. This research addresses the need for an objective, continuous, and non-disruptive measure of mental workload.

Conceptual overview of the experimental setup, including the EEG cap, CubTile input device, and the 3D bridge-building task.

Methodology

To build a robust workload estimator, the researchers combined brain activity (EEG) with physiological signals, specifically ElectroCardioGraphy (ECG) for heart rate and Galvanic Skin Response (GSR) for skin conductance.

Calibration and the Context Shift

The system was first calibrated using a standard N-back task to define "high" and "low" workload states for each participant. A significant hurdle in Brain-Computer Interface (BCI) research is the "context shift": brain signals recorded while a user is sitting still (calibration) differ significantly from those recorded during active physical movement (3D manipulation). To overcome this, the authors employed Regularized Common Spatial Patterns (rCSP), a technique that adjusts EEG filters to remain robust against the signal noise introduced by physical activity.

The 3D Manipulation Task

Participants utilized the CubTile—a five-sided multi-touch cubic interface—to perform a virtual bridge-building task. This involved seven distinct docking maneuvers. To validate the workload measure, the researchers included a "sabotage" task where rotation and translation controls were secretly inverted, intentionally spiking the user's mental effort.

The experimental environment showing the CubTile multi-touch interface and the virtual bridge reconstruction task.

Key Results and Insights

The study, involving 8 participants, yielded several significant findings:

  • High Classification Accuracy: The system achieved an average accuracy of 88.6% in discriminating between high and low workload states using EEG signals. Interestingly, including high-frequency bands (which likely captured some muscle activity or EMG) improved performance.
  • The "Inversion" Spike: The workload index clearly peaked during the task with inverted controls. This confirms that the system measures genuine mental struggle rather than just general physical activity.
  • Physiological Signal Limitations: In this specific study, ECG and GSR performed near chance levels. The authors attribute this to the short analysis windows (2–10 seconds); physiological signals typically require longer durations (30 seconds to 2 minutes) to reflect meaningful trends.
  • Measurable Learning Curves: The data suggests that such systems can track how a user "learns" an interface. A drop in workload over time while maintaining high performance is a clear indicator of skill acquisition.

Discussion and Future Work

While promising, this research is a proof-of-concept with current limitations. The analysis was performed offline, meaning the algorithm currently requires data from the entire session to function. Future work will focus on real-time estimation to provide live feedback to UI designers.

Additionally, the current hardware setup—involving 30 EEG electrodes—is not yet practical for standard corporate UX testing. However, as sensor technology becomes more ergonomic, this approach offers a path toward a "heat map" of mental effort, allowing designers to identify specific interaction bottlenecks that traditional surveys might miss. Beyond workload, future iterations could explore other mental states such as frustration or boredom to provide a holistic view of the user experience.

Summary updated

BibTeX
@inproceedings{wobrock2015continuous,
  author = {Dennis Wobrock and Jérémy Frey and Delphine Graeff and Jean-Baptiste de la Rivière and Julien Castet and Fabien Lotte},
  title = {{Continuous Mental Effort Evaluation During 3D Object Manipulation Tasks Based on Brain and Physiological Signals}},
  booktitle = {Interact},
  year = {2015},
  doi = {10.1007/978-3-319-22701-6\_35},
}
Figure