TL;DR
TOMAS is a novel framework for predicting future hand movements in first-person (egocentric) videos by jointly reasoning about hand skeletal structure and object semantics. By breaking down complex actions into "adaptive stages" with learnable temporal boundaries, it significantly outperforms existing methods in forecasting realistic, intent-driven hand trajectories.
What Problem Does This Paper Address?
Imagine you are watching a first-person video of someone reaching for a cup. Are they going to pick it up, push it aside, or pull it closer? Even if you know exactly where the cup is, the hand's path will look completely different depending on the intended action.
Existing hand-forecasting models often struggle with this interaction ambiguity. They might predict a "generic" straight line toward an object, failing to capture the nuanced "operation-dependent" motion patterns humans naturally exhibit. Furthermore, actions aren't just single states; they are sequences of phases (like "approach," "pre-grasp," and "manipulate") that vary in duration and spatial requirements. TOMAS aims to bridge this gap by explicitly modeling the relationship between hand topology, object semantics, and the temporal stages of an action.
Methods
TOMAS (Topology-Aware Operation Modeling and Adaptive Staging) introduces three key innovations:
- Topology-Aware Reasoning: Instead of treating the hand as a single point, TOMAS uses a "Hand Topology Encoder" to process 21 articulated joints and their skeletal connections. It combines this with object features (using CLIP and Mask R-CNN) to infer both the target object and the latent "operation" (e.g., pinching vs. lifting). This results in interaction-conditioning tokens that guide the rest of the model.
- Adaptive Staging Mechanism: Rather than assuming every action follows a fixed timeline, TOMAS organizes future motion into ordered latent stages. It uses a Mamba-based temporal backbone to predict "duration scores" for these stages, allowing the model to learn where one phase of an action ends and the next begins based on the specific context.
- Geometric Subgoals: For each stage, the model predicts a geometric subgoal—an intermediate spatial target. These subgoals act as anchors for a hierarchical trajectory decoder, ensuring the predicted path isn't just accurate at the endpoint, but follows a realistic "arc" throughout the movement.
Datasets
The authors evaluated TOMAS on three major egocentric vision benchmarks:
- EgoPAT3D-DT: Supports both 2D and 3D trajectory forecasting with specific splits for "seen" and "unseen" environments.
- H2O-PT: A dataset focused on two-hand manipulation, used here for 2D and 3D forecasting.
- EGTEA Gaze+: A large-scale dataset used for short-horizon 2D hand trajectory prediction.
Key Insights and Ideas
- Hand Shape is a "Pre-cue": The way a hand starts to shape itself (its topology) even before touching an object is a massive hint about the intended operation. TOMAS is one of the first to deeply integrate this skeletal "pose-intent" into the forecasting pipeline.
- Temporal Flexibility Matters: Actions are "nonuniform." A quick push has different temporal dynamics than a careful grasp. By making stage boundaries learnable (adaptive), TOMAS can stretch or compress phases of the trajectory to match the inferred intent.
- Subgoals vs. Waypoints: Instead of forcing the hand to pass through rigid points, the stage-wise subgoals provide "continuous geometric guidance." This allows the trajectory decoder to maintain smoothness while still being anchored to the necessary intermediate spatial objectives.
Key Results
TOMAS achieved state-of-the-art (SOTA) performance across virtually all tested metrics:
- 3D Accuracy: On the H2O-PT benchmark, TOMAS reduced the Mean Average Displacement Error (ADE) to 0.025m, a significant improvement over the previous SOTA of 0.030m.
- Generalization: In "unseen" environments on EgoPAT3D-DT, TOMAS maintained a strong lead, showing that its reasoning about hand-object relationships generalizes well to new scenes.
- Qualitative Realism: Visualizations show that TOMAS predicts natural, curved approach paths that account for the object's geometry, whereas baseline models often produce "robotic" straight lines that ignore the physical context of the interaction.
Limitations and Future Work
While TOMAS is highly effective, the authors note a few areas for growth:
- Object Dependency: The model's performance is tied to the quality of the candidate object proposals. If an object isn't detected or tracked well in the "past" frames, the "future" reasoning suffers.
- Single-Hand Focus: The current framework is primarily optimized for single-hand forecasting. Extending this to complex bimanual (two-handed) tasks where hands might coordinate or interfere with each other is a logical next step.
- Long-Term Planning: The current prediction horizon is relatively short (around 0.5 to 1.5 seconds). Future work could explore how these "stages" might scale to multi-step tasks like cooking or assembly.
Summary updated