Convolutional Long Short-Term Memory Networks for Recognizing First Person Interactions

Swathikiran Sudhakaran, Oswald Lanz

Abstract

In this paper, we present a novel deep learning based approach for addressing the problem of interaction recognition from a first person perspective. The proposed approach uses a pair of convolutional neural networks, whose parameters are shared, for extracting frame level features from successive frames of the video. The frame level features are then aggregated using a convolutional long short-term memory. The hidden state of the convolutional long short-term memory, after all the input video frames are processed, is used for classification in to the respective categories. The two branches of the convolutional neural network perform feature encoding on a short time interval whereas the convolutional long short term memory encodes the changes on a longer temporal duration. In our network the spatio-temporal structure of the input is preserved till the very final processing stage. Experimental results show that our method outperforms the state of the art on most recent first person interactions datasets that involve complex ego-motion. In particular, on UTKinect-FirstPerson it competes with methods that use depth image and skeletal joints information along with RGB images, while it surpasses all previous methods that use only RGB images by more than 20% in recognition accuracy.

Introduction: The Challenge of Egocentric Interaction

Recognizing human-to-human interactions from a first-person perspective (e.g., shaking hands, receiving an object, or hugging) presents unique challenges compared to standard third-person action recognition. Egocentric video is characterized by significant "ego-motion"—the constant jitter and shifting caused by the wearer's head or body movements. This motion often obscures the actual interaction and can confuse standard tracking algorithms. Furthermore, interactants frequently move in and out of the narrow field of view as they approach the observer.

Historically, researchers relied on hand-crafted features like optical flow or skeletal joint tracking from specialized sensors (e.g., Kinect). This paper introduces an end-to-end deep learning architecture specifically designed to overcome these hurdles using only raw RGB video frames, successfully replacing manual pipelines with a unified model.

Proposed Architecture: Preserving Spatial Context

The core innovation of this work is an architecture that maintains the spatial structure of video features throughout the entire processing pipeline. Most previous video architectures flatten spatial features into a 1D vector before temporal processing, which loses crucial context for distinguishing between the wearer's motion and the interactant's motion.

Detailed architecture of the proposed model showing the Siamese CNN branches and the ConvLSTM aggregator.

The model consists of three primary stages:

  1. Siamese Feature Extraction: Two branches of a pre-trained AlexNet (with fully-connected layers removed) share parameters to extract spatial features from successive video frames.
  2. Spatio-Temporal Aggregation: Instead of flattening features, the model passes them through a 3D Convolutional layer followed by a Convolutional Long Short-Term Memory (convLSTM). By using convolutional gates, the convLSTM "remembers" spatial locations of objects while tracking their temporal evolution.
  3. Efficient Classification: To prevent overfitting on small egocentric datasets, the model replaces traditional dense layers with a final spatial convolution followed by Global Average Pooling (GAP) and a Softmax layer. This reduces the parameter count significantly (21.8M vs. 61.3M in standard LRCN models).

The researchers evaluated two input modalities: raw frames and difference images (pixel-wise subtraction of consecutive frames). They found that difference images excel in stable environments (like the JPL dataset), while raw frames are more robust to high ego-motion (like the UTKinect dataset), as the network learns to filter out camera noise and focus on object context.

Experimental Evaluation

The model was tested on four established first-person interaction benchmarks: the JPL First-Person Interaction Dataset (JPLFPID), the NUS First Person Interaction Dataset (NUSFPID), and two subsets of the UTKinect-FirstPerson dataset (Humanoid and Non-humanoid).

Comparison of the proposed method against state-of-the-art techniques across multiple egocentric datasets.

As shown in the results table, the proposed method set new benchmarks across the board:

  • UTKinect-FirstPerson: On RGB-only data, the model surpassed previous state-of-the-art methods by over 20% in recognition accuracy.
  • Multimodal Competitiveness: Remarkably, the RGB-only model performed comparably to, and in some cases better than, older methods that had access to extra data like depth maps and 3D skeletal joints.
  • SOTA on NUSFPID: The model achieved a new high of 70.0% accuracy, a substantial improvement over the previous best of 61.8%.

Conclusion

By integrating Siamese CNNs with convLSTM units, this research demonstrates that preserving spatial information is critical for egocentric interaction recognition. The architecture's efficiency and robustness to ego-motion set a new standard for the field. Future work may explore joint training regimes that adaptively combine raw pixels and motion differences to further enhance performance across varying levels of camera stability, as well as handling longer-term temporal dependencies beyond 20 frames.

Summary updated

Abstract

In this paper, we present a novel deep learning based approach for addressing the problem of interaction recognition from a first person perspective. The proposed approach uses a pair of convolutional neural networks, whose parameters are shared, for extracting frame level features from successive frames of the video. The frame level features are then aggregated using a convolutional long shortterm memory. The hidden state of the convolutional long short-term memory, after all the input video frames are processed, is used for classification in to the respective categories. The two branches of the convolutional neural network perform feature encoding on a short time interval whereas the convolutional long short term memory encodes the changes on a longer temporal duration. In our network the spatio-temporal structure of the input is preserved till the very final processing stage. Experimental results show that our method outperforms the state of the art on most recent first person interactions datasets that involve complex ego-motion. In particular, on UTKinect-FirstPerson it competes with methods that use depth image and skeletal joints information along with RGB images, while it surpasses all previous methods that use only RGB images by more than 20% in recognition accuracy.
BibTeX
@inproceedings{sudhakaran2017convolutional,
  author = {Swathikiran Sudhakaran and Oswald Lanz},
  title = {{Convolutional Long Short-Term Memory Networks for Recognizing First Person Interactions}},
  booktitle = {Workshops @ ICCV},
  year = {2017},
  url = {https://openreview.net/forum?id=TaUxJZYzxLE},
}
Figure