Trespassing the Boundaries: Labeling Temporal Bounds for Object Interactions in Egocentric Video

Davide Moltisanti, Michael Wray, Walterio W. Mayol-Cuevas, Dima Damen

Abstract

Action recognition in egocentric video relies heavily on the precise temporal grounding of interactions. However, current datasets suffer from significant inconsistencies in how human annotators define the start and end points of actions. This paper investigates the impact of these temporal boundary inconsistencies on state-of-the-art models and proposes a new labeling protocol: Rubicon Boundaries (RB). Inspired by cognitive psychology, the RB protocol provides a consistent framework for temporal annotation, leading to a substantial increase in annotator agreement (IoU from 0.63 to 0.83) and a 4% boost in action recognition accuracy.

Introduction

In computer vision, identifying what occurs in a video is often prioritized over when it occurs. For egocentric video—where interactions are often messy and continuous—defining the exact temporal boundaries of an action like "opening a fridge" is subjective. Does the action begin when the hand moves, or when it touches the handle? These minor labeling discrepancies create "temporal noise" that acts as a bottleneck for modern recognition models.

The Problem: Temporal Inconsistency

Action recognition models are typically trained on trimmed video clips. The researchers found that existing datasets are riddled with inconsistencies between "tight" annotators (focusing only on physical contact) and "loose" annotators (including approach and follow-through). Because deep learning models, particularly Two-Stream CNNs, are highly sensitive to the specific frames seen during training, this noise leads to unreliable performance and evaluation.

Methods and Robustness Testing

The study proceeded in three stages:

  1. Inconsistency Analysis: Quantifying annotator disagreement across the BEOID, GTEA Gaze+, and CMU Kitchen datasets.
  2. Robustness Testing: Systematically perturbing (shifting) the start and end times of action segments to measure the sensitivity of Improved Dense Trajectories (IDT) and Two-Stream CNN models.
  3. The Rubicon Proposal: Introducing a framework based on the Rubicon Model of Action Phases, which distinguishes between the Pre-actional Phase (preparatory motion) and the Actional Phase (core interaction).
Impact of temporal boundary shifts on recognition accuracy.

As shown in the robustness testing, even when a shifted segment maintains a 50% overlap with the ground truth, recognition accuracy can drop significantly. This suggests that the "fringes" of an action contain critical discriminative information that models rely on for correct classification.

Key Insights: The "Pre-actional" Boost

A major finding of the Rubicon protocol is that including the "reaching" or pre-actional phase often improves model performance. This phase provides essential context regarding the user's intent, helping the model differentiate between visually similar actions, such as "taking a cup" versus "washing a cup."

Key Results

The implementation of Rubicon Boundaries yielded two primary benefits:

  • Increased Consistency: The average Intersection-over-Union (IoU) between different human annotators jumped from 0.63 to 0.83.
  • Accuracy Gains: Re-annotating the GTEA Gaze+ dataset with the RB protocol resulted in a 4% overall accuracy increase for Two-Stream CNNs, with improvements seen in 55% of action classes without any changes to the model architecture.
Comparison of annotator agreement (IoU) between conventional and Rubicon Boundary (RB) annotations.

Conclusion and Future Work

By grounding temporal labeling in psychological principles, the Rubicon Boundaries protocol addresses a fundamental source of noise in egocentric video datasets. While the method significantly improves consistency among expert annotators, challenges remain in scaling this to crowdsourced platforms where workers may find the definitions less intuitive. Future research will explore the application of RB to more modern architectures, such as Transformers, and address the ongoing challenge of action granularity.

Summary updated

Abstract

Manual annotations of temporal bounds for object interactions (i.e. start and end times) are typical training input to recognition, localization and detection algorithms. For three publicly available egocentric datasets, we uncover inconsistencies in ground truth temporal bounds within and across annotators and datasets. We systematically assess the robustness of state-of-the-art approaches to changes in labeled temporal bounds, for object interaction recognition. As boundaries are trespassed, a drop of up to 10% is observed for both Improved Dense Trajectories and Two-Stream Convolutional Neural Network. We demonstrate that such disagreement stems from a limited understanding of the distinct phases of an action, and propose annotating based on the Rubicon Boundaries, inspired by a similarly named cognitive model, for consistent temporal bounds of object interactions. Evaluated on a public dataset, we report a 4% increase in overall accuracy, and an increase in accuracy for 55% of classes when Rubicon Boundaries are used for temporal annotations.
BibTeX
@inproceedings{moltisanti2017trespassing,
  author = {Davide Moltisanti and Michael Wray and Walterio W. Mayol-Cuevas and Dima Damen},
  title = {{Trespassing the Boundaries: Labeling Temporal Bounds for Object Interactions in Egocentric Video}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2017},
  doi = {10.1109/iccv.2017.314},
}
Figure