TOMAS: Topology-Aware Operation Modeling and Adaptive Staging for Egocentric Hand Forecasting

Guanzhong Sun, Junyi Ma, Yuxuan Wu, Erhang Zhang, Chenyang Xu, Yanzi Miao, Hesheng WangarXiv 2026 · arXiv

TL;DR

TOMAS is a novel framework for predicting future hand movements in first-person (egocentric) videos by jointly reasoning about hand skeletal structure and object semantics. By breaking down complex actions into "adaptive stages" with learnable temporal boundaries, it significantly outperforms existing methods in forecasting realistic, intent-driven hand trajectories.

What Problem Does This Paper Address?

Imagine you are watching a first-person video of someone reaching for a cup. Are they going to pick it up, push it aside, or pull it closer? Even if you know exactly where the cup is, the hand's path will look completely different depending on the intended action.

Existing hand-forecasting models often struggle with this interaction ambiguity. They might predict a "generic" straight line toward an object, failing to capture the nuanced "operation-dependent" motion patterns humans naturally exhibit. Furthermore, actions aren't just single states; they are sequences of phases (like "approach," "pre-grasp," and "manipulate") that vary in duration and spatial requirements. TOMAS aims to bridge this gap by explicitly modeling the relationship between hand topology, object semantics, and the temporal stages of an action.

Fig. 1. Operation-dependent hand forecasting. The same object can support different operations with distinct future hand trajectories.

Methods

TOMAS (Topology-Aware Operation Modeling and Adaptive Staging) introduces three key innovations:

  1. Topology-Aware Reasoning: Instead of treating the hand as a single point, TOMAS uses a "Hand Topology Encoder" to process 21 articulated joints and their skeletal connections. It combines this with object features (using CLIP and Mask R-CNN) to infer both the target object and the latent "operation" (e.g., pinching vs. lifting). This results in interaction-conditioning tokens that guide the rest of the model.
  2. Adaptive Staging Mechanism: Rather than assuming every action follows a fixed timeline, TOMAS organizes future motion into ordered latent stages. It uses a Mamba-based temporal backbone to predict "duration scores" for these stages, allowing the model to learn where one phase of an action ends and the next begins based on the specific context.
  3. Geometric Subgoals: For each stage, the model predicts a geometric subgoal—an intermediate spatial target. These subgoals act as anchors for a hierarchical trajectory decoder, ensuring the predicted path isn't just accurate at the endpoint, but follows a realistic "arc" throughout the movement.
Fig. 2. System overview of TOMAS. Given observed egocentric video and historical hand trajectories, TOMAS employs topology-aware hand–object reasoning to jointly model candidate-object semantics and latent operation representations. The resulting interaction-aware representations guide ordered latent motion stages and stage-wise geometric subgoals for future hand trajectory forecasting.

Datasets

The authors evaluated TOMAS on three major egocentric vision benchmarks:

  • EgoPAT3D-DT: Supports both 2D and 3D trajectory forecasting with specific splits for "seen" and "unseen" environments.
  • H2O-PT: A dataset focused on two-hand manipulation, used here for 2D and 3D forecasting.
  • EGTEA Gaze+: A large-scale dataset used for short-horizon 2D hand trajectory prediction.

Key Insights and Ideas

  • Hand Shape is a "Pre-cue": The way a hand starts to shape itself (its topology) even before touching an object is a massive hint about the intended operation. TOMAS is one of the first to deeply integrate this skeletal "pose-intent" into the forecasting pipeline.
  • Temporal Flexibility Matters: Actions are "nonuniform." A quick push has different temporal dynamics than a careful grasp. By making stage boundaries learnable (adaptive), TOMAS can stretch or compress phases of the trajectory to match the inferred intent.
  • Subgoals vs. Waypoints: Instead of forcing the hand to pass through rigid points, the stage-wise subgoals provide "continuous geometric guidance." This allows the trajectory decoder to maintain smoothness while still being anchored to the necessary intermediate spatial objectives.

Key Results

TOMAS achieved state-of-the-art (SOTA) performance across virtually all tested metrics:

  • 3D Accuracy: On the H2O-PT benchmark, TOMAS reduced the Mean Average Displacement Error (ADE) to 0.025m, a significant improvement over the previous SOTA of 0.030m.
  • Generalization: In "unseen" environments on EgoPAT3D-DT, TOMAS maintained a strong lead, showing that its reasoning about hand-object relationships generalizes well to new scenes.
  • Qualitative Realism: Visualizations show that TOMAS predicts natural, curved approach paths that account for the object's geometry, whereas baseline models often produce "robotic" straight lines that ignore the physical context of the interaction.

Limitations and Future Work

While TOMAS is highly effective, the authors note a few areas for growth:

  • Object Dependency: The model's performance is tied to the quality of the candidate object proposals. If an object isn't detected or tracked well in the "past" frames, the "future" reasoning suffers.
  • Single-Hand Focus: The current framework is primarily optimized for single-hand forecasting. Extending this to complex bimanual (two-handed) tasks where hands might coordinate or interfere with each other is a logical next step.
  • Long-Term Planning: The current prediction horizon is relatively short (around 0.5 to 1.5 seconds). Future work could explore how these "stages" might scale to multi-step tasks like cooking or assembly.

Summary updated

Abstract

Forecasting future hand movements from egocentric videos plays an important role in understanding human intentions and hand-object interactions. However, different manipulation intentions can lead to substantially different hand trajectories even when interacting with the same object. Existing methods have not sufficiently explored the intrinsic relationships between articulated hand movements and object semantics to capture such operation-dependent motion patterns. Therefore, we propose TOMAS, a topology-aware operation modeling and adaptive staging framework for egocentric hand forecasting. Specifically, we develop a topology-aware hand-object reasoning mechanism that associates hand skeletal structures and articulation dynamics with object semantics and spatial relationships to jointly infer potential interaction targets and latent manipulation operations. The resulting object-operation representations provide interaction conditions for future hand trajectory prediction, with supervision adapted to the annotations available in each dataset. Furthermore, we introduce an adaptive staging mechanism that organizes future hand movements into ordered latent stages with learnable temporal boundaries. Stage-specific geometric subgoals are then incorporated to guide trajectory refinement according to the underlying manipulation patterns. Experimental results on multiple publicly available benchmarks demonstrate that TOMAS achieves state-of-the-art performance in multi-dimensional and multi-target hand motion forecasting.
BibTeX
@misc{sun2026tomas,
  author = {Guanzhong Sun and Junyi Ma and Yuxuan Wu and Erhang Zhang and Chenyang Xu and Yanzi Miao and Hesheng Wang},
  title = {{TOMAS: Topology-Aware Operation Modeling and Adaptive Staging for Egocentric Hand Forecasting}},
  year = {2026},
  eprint = {2611.23970},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23970},
}
Figure