Towards an Unequivocal Representation of Actions

Michael Wray, Davide Moltisanti, Dima Damen

TL;DR

This paper introduces "Soft Assigned Multi-Label" (SAML) representations for actions, moving beyond single-verb labels to capture the inherent ambiguity and overlapping nature of human actions. By assigning a soft score to a richer set of verbs, SAML improves action recognition and retrieval accuracy, offering a more nuanced understanding of egocentric interactions.

What Problem Does This Paper Address?

Describing human actions, especially in egocentric videos (first-person perspective), is challenging due to inherent semantic ambiguities. For instance, the motion of "opening" can apply to a door, a cupboard, or a bottle, and while objects help disambiguate in verb-noun labels (e.g., "open door"), this ties actions to specific objects. This can be problematic when the same motion is applied to different objects (e.g., opening a cupboard vs. a microwave). Current approaches often rely on single-verb labels, which struggle with these overlaps and can lead to confusion for human annotators. The paper aims to create an "Unequivocal Representation of Actions" that can handle these ambiguities and generalize better across different objects.

Methods

The core contribution is the Soft Assigned Multi-Label (SAML) representation. Unlike traditional single-verb labels (SL) or hard multi-verb labels (ML), SAML assigns a continuous score between 0 and 1 to each verb in a large vocabulary, indicating its relevance to a given action segment. This "soft assignment" allows for a more nuanced representation, accommodating sub-actions (like "hold" when pouring) and capturing the degree to which a verb describes an interaction.

The authors train a two-stream fusion Convolutional Neural Network (CNN) using this SAML representation. The learning objective is formulated as a multi-output regression problem, employing a sigmoid binary cross-entropy loss. This approach avoids independence assumptions between verbs and aims to approximate the soft assignment scores directly.

For evaluation, they propose two metrics:

  1. Accuracy (A(αØ)): Measures the overlap between predicted and ground-truth verbs above a certain relevance threshold (α).
  2. Retrieval: Evaluates the ability to retrieve videos based on text queries (text-to-video) or retrieve videos based on other videos (video-to-video), using L2 distance in the learned embedding space.

Datasets

The paper evaluates its approach on three publicly available egocentric datasets:

  • BEOID (732 video segments)
  • CMU (404 video segments)
  • GTEA+ (1001 video segments)

These datasets are annotated with verb-noun pairs or, in some cases, multiple verbs. The authors leverage existing annotations and adapt them for their multi-verb labeling schemes.

Key Insights and Ideas

  • Action ambiguity is inherent and can be embraced: Instead of trying to eliminate ambiguity with object labels, SAML embraces it by allowing multiple verbs to describe an action with varying degrees of relevance.
  • Soft assignment is crucial for sub-actions: Verbs like "hold" might not fully describe an interaction but are essential context. Soft assignment allows these less dominant but relevant verbs to be represented without being ignored.
  • Richer verb vocabulary improves understanding: By increasing the pool of verbs and allowing soft assignments, the model can learn a more expressive representation that captures finer-grained distinctions.
  • Egocentric domain benefits most: The paper focuses on egocentric videos where interactions are frequent and successive, making nuanced action representation particularly valuable.

Key Results

  • Action Recognition: SAML consistently outperforms single-verb labels (SL) across all three datasets. It achieves comparable results to verb-noun (V-N) labels on BEOID and CMU, and even outperforms V-N on the largest dataset, GTEA+ (achieving 67.8% accuracy with SAML vs. 61.2% with V-N).
  • Video-to-Text Retrieval: SAML consistently achieves the highest average Mean Average Precision (mAP) for retrieval, outperforming both SL and ML, especially when considering higher relevance thresholds (e.g., α ≥ 0.5 ranking).
  • Text-to-Video Retrieval: SAML shows improved performance as the number of verbs in the query increases, suggesting its ability to leverage the full vocabulary for more complex queries.
  • Cross-Dataset Retrieval: Qualitative examples demonstrate SAML's ability to retrieve semantically similar actions across different datasets, even when the specific objects or verb combinations differ (e.g., "pull drawer" from BEOID retrieving "open freezer" from GTEA+).

Limitations and Future Work

  • Semantic relationships are challenging: The paper notes that explicitly stating semantic relationships between verbs (synonyms, context, sub-actions) is difficult and not readily available in lexical databases. While they explore WordNet and Word2Vec, their limitations are highlighted.
  • Future Work: The authors plan to explore zero-shot or few-shot learning by predicting novel combinations of verb labels, leveraging the learned rich representations. They also suggest assessing the learned representations further.

Summary updated

Abstract

This work introduces verb-only representations for actions and interactions; the problem of describing similar motions (e.g. 'open door', 'open cupboard'), and distinguish differing ones (e.g. 'open door' vs 'open bottle') using verb-only labels. Current approaches for action recognition neglect legitimate semantic ambiguities and class overlaps between verbs (Fig. 1), relying on the objects to disambiguate interactions. We deviate from single-verb labels and introduce a mapping between observations and multiple verb labels - in order to create an Unequivocal Representation of Actions. The new representation benefits from increased vocabulary and a soft assignment to an enriched space of verb labels. We learn these representations as multi-output regression, using a two-stream fusion CNN. The proposed approach outperforms conventional single-verb labels (also known as majority voting) on three egocentric datasets for both recognition and retrieval.
BibTeX
@inproceedings{wray2018towards,
  author = {Michael Wray and Davide Moltisanti and Dima Damen},
  title = {{Towards an Unequivocal Representation of Actions}},
  booktitle = {Workshops @ CVPR},
  year = {2018},
}
Figure