Visualizing Causality in Mixed Reality for Manual Task Learning: A Study

Rahul Jain, Jingyu Shi, Andrew Benton, Moiz Rasheed, Hyungjun Doh, Subramanian Chidambaram, Karthik Ramani

TL;DR

This paper explores how visualizing causal relationships in Mixed Reality (MR) can improve manual task learning. They found that while showing causality increases learning time, it leads to better task performance and fewer tutorial revisits, suggesting a trade-off between learning efficiency and depth of understanding. Specifically, visualizing gesture-level causality provided the best post-learning performance.

What Problem Does This Paper Address?

Learning complex manual tasks, especially those requiring fine motor skills and hand-eye coordination, can be challenging. While Mixed Reality (MR) offers immersive and in-situ learning environments, current MR-based training often focuses on presenting only the immediate steps of a task. This approach might not equip learners with a deep understanding of why each step is necessary or how it leads to subsequent actions and the overall goal. The paper addresses this gap by investigating how visualizing different levels of causal relationships (event, interaction, and gesture) in MR impacts user comprehension, learning gain, and task execution.

Methods

The researchers developed a novel MR system that visualizes causality in manual tasks. They proposed a three-level hierarchy for causality:

  • Event-level causality: Links between distinct events in a task.
  • Interaction-level causality: Links between specific actions within an event.
  • Gesture-level causality: Links between fine-grained hand movements and object manipulations.

Their MR system presents users with demonstrations of the current step, along with causally related future steps at one or more of these levels. They conducted a user study with 48 participants, divided into four groups:

  • Control Group (GC): Saw only the current step's demonstration.
  • Group G1: Saw current step + event-level causality.
  • Group G2: Saw current step + event-level and interaction-level causality.
  • Group G3: Saw current step + all three levels of causality (event, interaction, gesture).

Participants learned a camera setup assembly task in a 1-hour learning phase and then performed the task independently in a testing phase. Performance was measured by learning time, testing time, number of mistakes, and tutorial revisits. Subjective feedback was collected via questionnaires and interviews.

Datasets

Not discussed in the paper. The study used a custom-built testbed and created 3D models of objects for the camera setup assembly task.

Key Insights and Ideas

  • Causality aids understanding, but at a cost: Visualizing causality significantly improves understanding of the task and leads to better performance in the testing phase, but it requires more time during the initial learning phase.
  • Fewer revisits, deeper learning: Participants who received causal information (especially G3) revisited tutorials less often, indicating they grasped the task more effectively without needing constant re-instruction.
  • Gesture-level detail is crucial for performance: While all levels of causality showed benefits, visualizing gesture-level causality (G3) resulted in the fastest completion time during the testing phase, suggesting that understanding the fine-grained movements is key for efficient execution.
  • Interaction-level offers efficiency: Group G2 (event + interaction causality) spent the least amount of time in the learning phase, suggesting a potential sweet spot for learning efficiency.
  • Mistakes are hard to differentiate: The study did not find significant differences in the number of mistakes made across groups, possibly due to the task's nature or limitations in the measurement of errors.

Key Results

  • Learning Time: G1 (event-level) spent the longest time learning (M=24.71 min), while G2 (event+interaction) spent the least (M=21.26 min). The control group (GC) spent the least time learning overall (M=18.24 min), but G3 (all levels) spent more time than GC (M=22.27 min), failing hypothesis H1.
  • Testing Time: G3 (all levels) had the significantly shortest testing time (M=4.31 min), supporting hypotheses H2 and H5.
  • Tutorial Revisits: GC revisited tutorials significantly more than G3 (3.17 vs. 2.33 times), indicating better learning with causal information.
  • Subjective Ratings: G3 participants reported significantly higher understanding of the task's importance and outcomes compared to GC.

Limitations and Future Work

  • Learning Time vs. Efficiency: The primary limitation is the increased learning time required when incorporating causal information. Future work should focus on developing more intuitive and efficient visualization techniques to mitigate this.
  • Error Measurement: The lack of significant differences in mistake numbers suggests that the current error measurement might not be sensitive enough or that the task was not complex enough to reveal these differences. More rigorous setups and metrics are needed.
  • Testbed Limitations: The study relied on object and gesture tracking, which can be affected by occlusion and blurriness. Future work could explore more robust tracking methods.
  • Dynamic Causality: The authors suggest exploring dynamic causality, where learners see how their actions directly influence future outcomes in real-time, potentially leading to deeper understanding.
  • Error Cascades: Visualizing how initial errors lead to subsequent mistakes could further enhance learning by highlighting the importance of each step.
  • Sample Size: While the study found significant effects, a larger sample size could provide greater statistical power and detect smaller effect sizes more reliably.

Summary updated

BibTeX
@article{jain2025visualizing,
  author = {Rahul Jain and Jingyu Shi and Andrew Benton and Moiz Rasheed and Hyungjun Doh and Subramanian Chidambaram and Karthik Ramani},
  title = {{Visualizing Causality in Mixed Reality for Manual Task Learning: A Study}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2025},
  doi = {10.1109/tvcg.2025.3542949},
}
Figure