Human-like Relational Models for Activity Recognition in Video

Joseph Chrol-Cannon, Andrew Gilbert, Ranko Lazić, Adithya Madhusoodanan, Frank Guérin

Abstract

Video activity recognition by deep neural networks is impressive for many classes. However, it falls short of human performance, especially for challenging to discriminate activities. Humans differentiate these complex activities by recognising critical spatio-temporal relations among explicitly recognised objects and parts, for example, an object entering the aperture of a container. Deep neural networks can struggle to learn such critical relationships effectively. Therefore we propose a more human-like approach to activity recognition, which interprets a video in sequential temporal phases and extracts specific relationships among objects and hands in those phases. Random forest classifiers are learnt from these extracted relationships. We apply the method to a challenging subset of the something-something dataset and achieve a more robust performance against neural network baselines on challenging activities.

Introduction

While state-of-the-art 3D Convolutional Neural Networks (CNNs) demonstrate high performance on general video recognition tasks, they frequently struggle with fine-grained discrimination between subtly different actions. For instance, distinguishing between "putting an object down" and "pretending to put it down" remains a significant challenge. These "bottom-up" models ingest raw pixels and often fail to capture the underlying logic of an action. In contrast, humans recognize activities by observing spatio-temporal relationships between objects. This research proposes a "top-down" relational model that focuses on these high-level interactions to improve recognition accuracy on complex, easily confused activities.

Methodology

The proposed pipeline moves away from end-to-end pixel learning, focusing instead on interpretable object-centric features and temporal phases.

Object-Centric Representation

The model discards raw pixel data, texture, and optical flow, relying solely on the bounding boxes of hands and principal objects. This sparse representation reduces visual noise and forces the model to focus on the geometric logic of the interaction.

Object-centric bounding boxes identifying the hand (cyan), the manipulated object (blue), and the target object (red).

Five-Phase Temporal Modeling

Every action is decomposed into five sequential temporal phases to capture the logical progression of the activity:

  1. Phase a: Initial state; object is present without manipulation.
  2. Phase b: Approach; the hand enters the scene.
  3. Phase c: Manipulation; the critical interaction occurs (e.g., placement).
  4. Phase d: Departure; the hand leaves the interaction zone.
  5. Phase e: Result; the final state is evident.
Visualization of Phase d, where the hand and object depart from the interaction area.

Relational Scoring and Classification

Handcrafted scoring functions rate how well frames match specific phases based on binary relations (e.g., Touching, Contained) and motion relations (e.g., MoveWithHand). Feature vectors derived from these scores (mean, median, max, min) are used to train binary Random Forest classifiers for each of the 17 action categories.

Experimental Results

The model was evaluated on a "hard" subset of the 20bn "something-something" dataset (v2), focusing on 17 related "Putting" and "Taking" actions.

Model mean Average Precision (mAP)
Baseline CNN 0.42
Proposed Relational Model 0.50
Fused Model (Proposed + CNN) 0.57

The relational model significantly outperformed the CNN baseline on this subset, particularly in distinguishing "pretending" actions where motion patterns are superficially similar but logically distinct.

Discussion and Conclusion

The relational approach offers superior interpretability, allowing for detailed error analysis. For example, "pretending" actions are often confused when the start and end phases (a and e) appear identical. The study also identified Phase b (hand approach) as a critical temporal anchor.

A primary limitation is the reliance on 2D bounding boxes, which can lead to ambiguity in spatial relations (e.g., "into" vs "behind") depending on the camera angle. Future work will explore the integration of 3D depth information and more sophisticated object recognition to better model physical constraints like containment.

Summary updated

Abstract

Video activity recognition by deep neural networks is impressive for many classes. However, it falls short of human performance, especially for challenging to discriminate activities. Humans differentiate these complex activities by recognising critical spatio-temporal relations among explicitly recognised objects and parts, for example, an object entering the aperture of a container. Deep neural networks can struggle to learn such critical relationships effectively. Therefore we propose a more human-like approach to activity recognition, which interprets a video in sequential temporal phases and extracts specific relationships among objects and hands in those phases. Random forest classifiers are learnt from these extracted relationships. We apply the method to a challenging subset of the something-something dataset and achieve a more robust performance against neural network baselines on challenging activities.
BibTeX
@misc{chrolcannon2021human,
  author = {Joseph Chrol-Cannon and Andrew Gilbert and Ranko Lazić and Adithya Madhusoodanan and Frank Guérin},
  title = {{Human-like Relational Models for Activity Recognition in Video}},
  year = {2021},
  eprint = {2107.05319},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.05319},
}
Figure