Abstract
In the vision of a “hybrid society,” the interaction between humans and embodied digital technologies is envisaged to be as smooth and seamless as human–human interaction. In the foreseeable future, however, humans will undoubtedly continue to interact with many machines through designated human–machine interfaces (HMIs). Designing such HMIs for production settings presents a particular challenge, as users may vary in their experience and expertise. Moreover, challenging issues of a specific HMI are often difficult to verbalize and therefore hard to obtain through user report and expert interviews alone. To make interactions through HMIs smooth, it is therefore crucial to evaluate HMIs with state-of-the-art technologies that do not require explicit report. Here, we demonstrate the use of eye tracking for HMI design in an industrial production setting, where smooth human–machine interaction is particularly critical to ensure safe and efficient operation. Using a real-life example, we illustrate how eye tracking allows dissociating users’ difficulties to find a particular interaction item (“search”) from their challenges in realizing that it is indeed the item to be operated (“verification”). We argue that this distinction is crucial for (re-) designing HMIs to optimize usability and that the usefulness of eye tracking extends beyond the specific context to human–machine interaction in general.
Introduction
As industrial machinery grows increasingly complex, the design of intuitive Human-Machine Interfaces (HMIs) has become a critical factor for both operational safety and production efficiency. Traditional UX/UI evaluation in manufacturing environments typically relies on post-task questionnaires, such as the System Usability Scale (SUS), or expert interviews. However, these methods are inherently limited by human memory, recency bias, and a potential reluctance among employees to admit confusion regarding their company's internal systems.
Furthermore, standard behavioral metrics—such as total task completion time—are often ambiguous. A delay in completing a task does not reveal why the user struggled. Did they fail to locate the correct button, or did they find it quickly but fail to understand its function? This research introduces a continuous, non-report evaluation method using mobile eye tracking to disambiguate these interaction bottlenecks in a real-world production environment.
Methodology
The study involved 30 male participants, ranging from novices to experts, who were equipped with Tobii Pro Glasses 2 mobile eye-tracking hardware. Participants were tasked with performing 20 standard operations on a physical industrial sorting machine's touch-screen HMI.
To process the resulting gaze data, the researchers developed an automated computer vision pipeline. This pipeline detects the HMI screen within the user's dynamic field of view and maps gaze coordinates directly to specific UI elements. This allows for the interaction to be decomposed into two distinct phases:
- Scanning Time: The duration from the start of the task until the user’s gaze first fixates on the target UI element.
- Verification Time: The duration from that initial fixation until the user executes the physical action (e.g., a button press).
This methodological split is essential for identifying whether a UI flaw is visual (lack of salience) or semantic (lack of clarity).
Key Insights and Results
Salience vs. Semantics
The study demonstrates that high scanning time indicates a lack of visual salience, suggesting the element needs to be larger, brighter, or better positioned. Conversely, high verification time indicates a lack of semantic clarity, meaning the label or icon is confusing; in such cases, simply making the button larger will not resolve the usability issue.
Expertise and Performance
As expected, experts significantly outperformed novices, completing tasks in 451 seconds with only 1.1 errors on average, compared to the novices' 837 seconds and 4.9 errors. This validation confirmed the experimental setup's ability to capture meaningful performance differences.
Case Studies in HMI Bottlenecks
- Case Study 1 (Visual Search): When searching for "cycle time," users exhibited high scanning times (e.g., 54.1s) but minimal verification times (1.3s). This clearly identified the information as being poorly placed rather than poorly labeled.
- Case Study 2 (The "Looked but Failed to See" Phenomenon): In a task requiring users to acknowledge a machine error, 11 participants failed. Eye tracking revealed that all 11 unsuccessful users looked directly at the correct "ACK" button for approximately one second but chose not to press it. Their average verification time was a staggering 68.9s. They found the button quickly (scanning time of 15.1s) but did not understand that "ACK" was the abbreviation for "Acknowledge Error."
Conclusion and Future Directions
Mobile eye tracking provides a granular level of insight that traditional behavioral data and subjective reports cannot reach. By separating scanning from verification, designers can target specific UI improvements—either visual or semantic—with high precision.
While real-world industrial environments present challenges like variable lighting and screen occlusion, the potential for this technology is significant. Future work may involve integrating eye tracking with Virtual Reality (VR) "digital twins" to refine HMIs before physical prototypes are built, or using pupillometry to monitor real-time cognitive load for adaptive, closed-loop interface designs.
Summary updated