Hand-Object Contact Transition Detection in Egocentric Video

Huy Anh Nguyen, Chang Dong, Feras Dayoub, Minh HoaiarXiv 2026 · arXiv

This paper, recently published by researchers at the Australian Institute for Machine Learning, introduces OASIS (Object-Aware Spotting for Interaction Sequences), a framework designed to pinpoint the exact frames when a hand touches and releases an object in first-person (egocentric) video. By jointly modeling both the hand and the object of interest, the authors achieve state-of-the-art precision in localizing these critical "contact transitions."

TL;DR

The authors propose OASIS, a unified framework that detects the precise start (touch) and end (release) of hand-object interactions in egocentric video by integrating object-aware features with hand-centric representations. Alongside the model, they introduce ContactTransition, a large-scale dataset with over 16,000 frame-accurate annotations, demonstrating that explicit object grounding is the key to solving the traditionally difficult problem of contact release detection.

What Problem Does This Paper Address?

In egocentric vision, understanding how humans manipulate the world is a foundational task. Most existing research treats "contact" as a per-frame state (is the hand touching something now?) or focuses only on the "touch" moment. However, a complete "contact episode" is defined by two boundaries: the touch (onset) and the release (offset).

Detecting these at frame-level precision is notoriously difficult. Egocentric video is plagued by rapid camera motion, motion blur, and frequent occlusions. While "touch" is often preceded by a clear approaching motion, "release" is much more subtle—the hand might remain hovering near the object, or the object might fall away while the hand stays still. Previous models that relied solely on hand motion struggled to distinguish a release from a simple change in grip.

Methods

The authors build upon a previous hand-centric model called HiCE but introduce several novel components to create OASIS:

  1. Object-of-Interest Branch: Instead of just looking at the hand, OASIS predicts a class-agnostic heatmap to localize the specific object being interacted with. This provides a "spatial anchor" for the model.
  2. Presence Gate: Not every frame contains a visible object. A "presence gate" (a small MLP) predicts an object-presence score to filter out noisy features when no candidate object is visible in the hand-centered crop.
  3. Transition-Aware Event Spotting: Standard action spotting often uses a softmax classifier, which forces different event types to compete. OASIS treats touch and release as independent classes scored against a shared background. This allows the model to handle bimanual interactions or very short contact episodes where touch and release occur in close proximity.
  4. Class-Specific Displacement: The model predicts separate temporal offsets for touch and release to refine the exact frame localization.
Fig. 1: Overview of OASIS. Starting from the hand-enhanced feature map produced by HiCE [37], OASIS predicts an object-of-interest heatmap and pools object-aware features from the predicted regions. A presence gate controls when object evidence should be used, and the resulting object-aware representation is fused into the temporal spotting stream. The final head predicts touch and release transitions with event scores and class-specific displacement refinement.

Datasets

The authors introduce ContactTransition, a benchmark specifically for this task. It is derived from existing egocentric datasets (HOI4D and TACO) but adds rigorous, manually verified frame-level annotations for both touch and release.

Table 1: Statistics for the ContactTransition dataset.

The dataset contains:

  • 4,021 egocentric videos.
  • 8,498 touch annotations.
  • 8,077 release annotations.
  • Object-of-interest supervision for training the heatmap branch.

Key Insights and Ideas

  • The Asymmetry of Contact: The "aha moment" here is that touch and release are not symmetric. Touch is motion-driven (hand approaching object), while release is relationship-driven (hand separating from object). This explains why hand-only models fail at release detection.
  • Object Grounding as an Anchor: By explicitly predicting where the object is, the model learns to detect the "sustained separation" that defines a release, even when hand pose changes are minimal.
  • Independent Scoring Matters: Moving away from mutually exclusive labels (softmax) to independent margins (sigmoid) significantly improved performance in complex bimanual scenes.

Key Results

OASIS consistently outperformed both two-stage and end-to-end baselines:

  • Overall Performance: OASIS achieved a mean Average Precision (mAP) of 51.19, surpassing the previous best (HiCE) by 3.61 points.
  • Release Detection: The gains were most dramatic here. On the strictest metric (AP@0, requiring exact frame matching), OASIS improved release detection by nearly 6 points over HiCE.
  • Downstream Utility: The authors tested OASIS as a "temporal anchor" for a Video Question Answering (VQA) task (HanDyVQA). Providing the model with predicted contact transitions improved reasoning about object parts and state changes, proving these transitions are a compact and useful representation of manipulation.

Limitations and Future Work

While OASIS is a significant step forward, the authors note a few areas for improvement:

  • Per-Hand Attribution: Currently, the model detects that a transition happened but doesn't explicitly attribute it to the left or right hand in the output (though it uses hand-centric features internally).
  • Object Identity: The model knows where the object is but not what it is. Integrating semantic object labels could further improve reasoning.
  • Complex Occlusions: Extremely cluttered scenes or cases where the hand completely masks the object during release remain challenging.

Summary updated

Abstract

This paper tackles the task of hand-object contact transition detection in egocentric video, which aims to localize the two temporal boundaries of a contact episode: the moment a hand first establishes contact with an object (touch) and the moment contact is released (release). We propose a unified framework that jointly models the interacting hand and object to accurately localize both transition types. The proposed framework augments hand-centric spatiotemporal representations with object-aware features learned directly from the video and introduces a transition-aware event spotting formulation tailored to contact transitions, without relying on external object or hand-object interaction detectors during inference. To support this task, we present ContactTransition, a dataset comprising 4,021 egocentric videos with 8,498 touch and 8,077 release annotations, together with object-of-interest supervision. Experiments demonstrate that the proposed method consistently outperforms strong two-stage and end-to-end event-spotting baselines on both transition types. Finally, we show that detected contact transitions provide effective temporal anchors for downstream egocentric hand-object reasoning.
BibTeX
@misc{nguyen2026hand,
  author = {Huy Anh Nguyen and Chang Dong and Feras Dayoub and Minh Hoai},
  title = {{Hand-Object Contact Transition Detection in Egocentric Video}},
  year = {2026},
  eprint = {2611.13486},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13486},
}
Figure