This paper, recently published by researchers at the Australian Institute for Machine Learning, introduces OASIS (Object-Aware Spotting for Interaction Sequences), a framework designed to pinpoint the exact frames when a hand touches and releases an object in first-person (egocentric) video. By jointly modeling both the hand and the object of interest, the authors achieve state-of-the-art precision in localizing these critical "contact transitions."
TL;DR
The authors propose OASIS, a unified framework that detects the precise start (touch) and end (release) of hand-object interactions in egocentric video by integrating object-aware features with hand-centric representations. Alongside the model, they introduce ContactTransition, a large-scale dataset with over 16,000 frame-accurate annotations, demonstrating that explicit object grounding is the key to solving the traditionally difficult problem of contact release detection.
What Problem Does This Paper Address?
In egocentric vision, understanding how humans manipulate the world is a foundational task. Most existing research treats "contact" as a per-frame state (is the hand touching something now?) or focuses only on the "touch" moment. However, a complete "contact episode" is defined by two boundaries: the touch (onset) and the release (offset).
Detecting these at frame-level precision is notoriously difficult. Egocentric video is plagued by rapid camera motion, motion blur, and frequent occlusions. While "touch" is often preceded by a clear approaching motion, "release" is much more subtle—the hand might remain hovering near the object, or the object might fall away while the hand stays still. Previous models that relied solely on hand motion struggled to distinguish a release from a simple change in grip.
Methods
The authors build upon a previous hand-centric model called HiCE but introduce several novel components to create OASIS:
- Object-of-Interest Branch: Instead of just looking at the hand, OASIS predicts a class-agnostic heatmap to localize the specific object being interacted with. This provides a "spatial anchor" for the model.
- Presence Gate: Not every frame contains a visible object. A "presence gate" (a small MLP) predicts an object-presence score to filter out noisy features when no candidate object is visible in the hand-centered crop.
- Transition-Aware Event Spotting: Standard action spotting often uses a softmax classifier, which forces different event types to compete. OASIS treats touch and release as independent classes scored against a shared background. This allows the model to handle bimanual interactions or very short contact episodes where touch and release occur in close proximity.
- Class-Specific Displacement: The model predicts separate temporal offsets for touch and release to refine the exact frame localization.
Datasets
The authors introduce ContactTransition, a benchmark specifically for this task. It is derived from existing egocentric datasets (HOI4D and TACO) but adds rigorous, manually verified frame-level annotations for both touch and release.
The dataset contains:
- 4,021 egocentric videos.
- 8,498 touch annotations.
- 8,077 release annotations.
- Object-of-interest supervision for training the heatmap branch.
Key Insights and Ideas
- The Asymmetry of Contact: The "aha moment" here is that touch and release are not symmetric. Touch is motion-driven (hand approaching object), while release is relationship-driven (hand separating from object). This explains why hand-only models fail at release detection.
- Object Grounding as an Anchor: By explicitly predicting where the object is, the model learns to detect the "sustained separation" that defines a release, even when hand pose changes are minimal.
- Independent Scoring Matters: Moving away from mutually exclusive labels (softmax) to independent margins (sigmoid) significantly improved performance in complex bimanual scenes.
Key Results
OASIS consistently outperformed both two-stage and end-to-end baselines:
- Overall Performance: OASIS achieved a mean Average Precision (mAP) of 51.19, surpassing the previous best (HiCE) by 3.61 points.
- Release Detection: The gains were most dramatic here. On the strictest metric (AP@0, requiring exact frame matching), OASIS improved release detection by nearly 6 points over HiCE.
- Downstream Utility: The authors tested OASIS as a "temporal anchor" for a Video Question Answering (VQA) task (HanDyVQA). Providing the model with predicted contact transitions improved reasoning about object parts and state changes, proving these transitions are a compact and useful representation of manipulation.
Limitations and Future Work
While OASIS is a significant step forward, the authors note a few areas for improvement:
- Per-Hand Attribution: Currently, the model detects that a transition happened but doesn't explicitly attribute it to the left or right hand in the output (though it uses hand-centric features internally).
- Object Identity: The model knows where the object is but not what it is. Integrating semantic object labels could further improve reasoning.
- Complex Occlusions: Extremely cluttered scenes or cases where the hand completely masks the object during release remain challenging.
Summary updated