Abstract
Designing 3D User Interfaces (UI) requires adequate evaluation tools to ensure good usability and user experience. While many evaluation tools are already available and widely used, existing approaches generally cannot provide continuous and objective measures of usability qualities during interaction without interrupting the user. In this paper, we propose to use brain (with ElectroEncephaloGraphy) and physiological (ElectroCardioGraphy, Galvanic Skin Response) signals to continuously assess the mental effort made by the user to perform 3D object manipulation tasks. We first show how this mental effort (a.k.a., mental workload) can be estimated from such signals, and then measure it on 8 participants during an actual 3D object manipulation task with an input device known as the CubTile. Our results suggest that monitoring workload enables us to continuously assess the 3DUI and/or interaction technique ease-of-use. Overall, this suggests that this new measure could become a useful addition to the repertoire of available evaluation tools, enabling a finer grain assessment of the ergonomic qualities of a given 3D user interface.
Introduction
Designing effective 3D User Interfaces (UIs) is a complex challenge. 3D interaction is inherently cognitively demanding, often requiring mental rotations and sophisticated spatial reasoning. Traditionally, designers have relied on "think-aloud" protocols, behavioral metrics like reaction time, or post-hoc questionnaires such as the NASA-TLX to evaluate these interfaces.
However, these traditional methods suffer from several critical flaws. Subjective questionnaires rely on user memory and can be biased by social pressure. "Think-aloud" protocols are disruptive, potentially altering the very mental workload they aim to measure. Furthermore, most existing tools lack granularity; they provide a single global score for a task but cannot pinpoint exactly when a user struggled during a multi-minute interaction. This research addresses the need for an objective, continuous, and non-disruptive measure of mental workload.
Methodology
To build a robust workload estimator, the researchers combined brain activity (EEG) with physiological signals, specifically ElectroCardioGraphy (ECG) for heart rate and Galvanic Skin Response (GSR) for skin conductance.
Calibration and the Context Shift
The system was first calibrated using a standard N-back task to define "high" and "low" workload states for each participant. A significant hurdle in Brain-Computer Interface (BCI) research is the "context shift": brain signals recorded while a user is sitting still (calibration) differ significantly from those recorded during active physical movement (3D manipulation). To overcome this, the authors employed Regularized Common Spatial Patterns (rCSP), a technique that adjusts EEG filters to remain robust against the signal noise introduced by physical activity.
The 3D Manipulation Task
Participants utilized the CubTile—a five-sided multi-touch cubic interface—to perform a virtual bridge-building task. This involved seven distinct docking maneuvers. To validate the workload measure, the researchers included a "sabotage" task where rotation and translation controls were secretly inverted, intentionally spiking the user's mental effort.
Key Results and Insights
The study, involving 8 participants, yielded several significant findings:
- High Classification Accuracy: The system achieved an average accuracy of 88.6% in discriminating between high and low workload states using EEG signals. Interestingly, including high-frequency bands (which likely captured some muscle activity or EMG) improved performance.
- The "Inversion" Spike: The workload index clearly peaked during the task with inverted controls. This confirms that the system measures genuine mental struggle rather than just general physical activity.
- Physiological Signal Limitations: In this specific study, ECG and GSR performed near chance levels. The authors attribute this to the short analysis windows (2–10 seconds); physiological signals typically require longer durations (30 seconds to 2 minutes) to reflect meaningful trends.
- Measurable Learning Curves: The data suggests that such systems can track how a user "learns" an interface. A drop in workload over time while maintaining high performance is a clear indicator of skill acquisition.
Discussion and Future Work
While promising, this research is a proof-of-concept with current limitations. The analysis was performed offline, meaning the algorithm currently requires data from the entire session to function. Future work will focus on real-time estimation to provide live feedback to UI designers.
Additionally, the current hardware setup—involving 30 EEG electrodes—is not yet practical for standard corporate UX testing. However, as sensor technology becomes more ergonomic, this approach offers a path toward a "heat map" of mental effort, allowing designers to identify specific interaction bottlenecks that traditional surveys might miss. Beyond workload, future iterations could explore other mental states such as frustration or boredom to provide a holistic view of the user experience.
Summary updated