Visualizing Causality in Mixed Reality for Manual Task Learning: An Exploratory Study

Rahul Jain, Jingyu Shi, Andrew Benton, Moiz Rasheed, Hyungjun Doh, Subramanian Chidambaram, Karthik Ramani

TL;DR

Standard Mixed Reality (MR) instructions often focus on the "what" (the current step) but ignore the "why" (how it affects the future). This study introduces a three-level hierarchy for visualizing intention-driven causality—event, interaction, and gesture—and finds that while showing these causal links increases initial learning time, it significantly improves task performance and user comprehension during execution.

Introduction

When learning manual tasks like assembly or repair in Mixed Reality (MR), most systems provide "rote" guidance—step-by-step instructions that learners follow without necessarily understanding the underlying logic. While efficient for immediate completion, this approach often fails to foster true skill acquisition. Learners may finish a task without grasping how a current action, such as tightening a specific bolt, impacts a later stage, like ensuring structural stability for a mounted component.

Human learning is naturally tied to causality. The authors address this gap by proposing a systematic framework for representing and visualizing cause-and-effect relationships in MR, moving beyond simple guidance toward effective skill learning.

The Intention-driven Causality Framework

The researchers developed a framework based on Intention-driven Causality, which decomposes tasks into a three-level hierarchy:

  1. Event Level: High-level goals (e.g., "Assemble the base").
  2. Interaction Level: Specific actions within an event (e.g., "Aligning the bolts").
  3. Gesture Level: Fine-grained hand-object affordances (e.g., "Grasping the bolt at a specific angle").
The three-level hierarchy of intention-driven causality: Event, Interaction, and Gesture levels.

By visualizing these levels, the system communicates the expert's intention to the learner, showing not just the "Cause" (current step) but also the "Effect" (the future step it enables).

System Implementation and Task Design

The system utilized an Oculus Quest 2 with a ZED Mini stereo camera for passthrough MR. A custom "Test Bed" was created to display a visualization graph of causal links. To evaluate the framework, the authors designed a complex Camera Assembly Task involving 5 major events and 24 interactions, requiring tools such as drills, hammers, and 3D-printed components.

The Mixed Reality interface (b-1) and its real-world application (b-2), showing the procedural graph and causal links.

Experimental Results

A user study with 48 participants compared four groups: a control group (GC) shown only the current step, and three groups (G1, G2, G3) shown increasing levels of causal information.

The Learning/Performance Trade-off

There is a clear "tax" on initial learning. Participants in the most informed group (G3) took longer to finish the learning phase (22.27 min) compared to the control group (18.24 min). However, this investment paid off during the memory-based testing phase:

  • Testing Speed: G3 completed the task in 4.31 minutes, significantly faster than GC's 5.20 minutes.
  • Instruction Dependency: Despite the increased information density, G3 participants revisited tutorials less frequently (2.33 times) than the control group (3.17 times).
  • Comprehension: G3 reported a significantly higher understanding of why actions were necessary (scoring ~4.4/5 vs ~2.6/5).

Discussion and Future Work

The study suggests that visualizing high-level causality alone (G1) is insufficient; without the "deeper" details of interactions and gestures, causal links are harder to internalize. By providing a multi-level view, MR systems can help users build more robust mental models.

Future work includes exploring Dynamic Modeling to show the adverse effects of mistakes (e.g., "If you don't tighten this now, the arm will fall off later") and developing AI-driven adaptive visualizations to manage cognitive load by showing causal links only when necessary.

Summary updated

BibTeX
@article{jain2025visualizing,
  author = {Rahul Jain and Jingyu Shi and Andrew Benton and Moiz Rasheed and Hyungjun Doh and Subramanian Chidambaram and Karthik Ramani},
  title = {{Visualizing Causality in Mixed Reality for Manual Task Learning: An Exploratory Study}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2025},
  doi = {10.1109/tvcg.2025.3542949},
}
Figure