Abstract
Action recognition in egocentric video relies heavily on the precise temporal grounding of interactions. However, current datasets suffer from significant inconsistencies in how human annotators define the start and end points of actions. This paper investigates the impact of these temporal boundary inconsistencies on state-of-the-art models and proposes a new labeling protocol: Rubicon Boundaries (RB). Inspired by cognitive psychology, the RB protocol provides a consistent framework for temporal annotation, leading to a substantial increase in annotator agreement (IoU from 0.63 to 0.83) and a 4% boost in action recognition accuracy.
Introduction
In computer vision, identifying what occurs in a video is often prioritized over when it occurs. For egocentric video—where interactions are often messy and continuous—defining the exact temporal boundaries of an action like "opening a fridge" is subjective. Does the action begin when the hand moves, or when it touches the handle? These minor labeling discrepancies create "temporal noise" that acts as a bottleneck for modern recognition models.
The Problem: Temporal Inconsistency
Action recognition models are typically trained on trimmed video clips. The researchers found that existing datasets are riddled with inconsistencies between "tight" annotators (focusing only on physical contact) and "loose" annotators (including approach and follow-through). Because deep learning models, particularly Two-Stream CNNs, are highly sensitive to the specific frames seen during training, this noise leads to unreliable performance and evaluation.
Methods and Robustness Testing
The study proceeded in three stages:
- Inconsistency Analysis: Quantifying annotator disagreement across the BEOID, GTEA Gaze+, and CMU Kitchen datasets.
- Robustness Testing: Systematically perturbing (shifting) the start and end times of action segments to measure the sensitivity of Improved Dense Trajectories (IDT) and Two-Stream CNN models.
- The Rubicon Proposal: Introducing a framework based on the Rubicon Model of Action Phases, which distinguishes between the Pre-actional Phase (preparatory motion) and the Actional Phase (core interaction).
As shown in the robustness testing, even when a shifted segment maintains a 50% overlap with the ground truth, recognition accuracy can drop significantly. This suggests that the "fringes" of an action contain critical discriminative information that models rely on for correct classification.
Key Insights: The "Pre-actional" Boost
A major finding of the Rubicon protocol is that including the "reaching" or pre-actional phase often improves model performance. This phase provides essential context regarding the user's intent, helping the model differentiate between visually similar actions, such as "taking a cup" versus "washing a cup."
Key Results
The implementation of Rubicon Boundaries yielded two primary benefits:
- Increased Consistency: The average Intersection-over-Union (IoU) between different human annotators jumped from 0.63 to 0.83.
- Accuracy Gains: Re-annotating the GTEA Gaze+ dataset with the RB protocol resulted in a 4% overall accuracy increase for Two-Stream CNNs, with improvements seen in 55% of action classes without any changes to the model architecture.
Conclusion and Future Work
By grounding temporal labeling in psychological principles, the Rubicon Boundaries protocol addresses a fundamental source of noise in egocentric video datasets. While the method significantly improves consistency among expert annotators, challenges remain in scaling this to crowdsourced platforms where workers may find the definitions less intuitive. Future research will explore the application of RB to more modern architectures, such as Transformers, and address the ongoing challenge of action granularity.
Summary updated