Unsupervised Learning of Deep Feature Representation for Clustering Egocentric Actions

Bharat Lal Bhatnagar, Suriya Singh, Chetan Arora, C. V. JawaharIJCAI 2017 · DOI

TL;DR

This paper introduces a novel unsupervised approach for clustering egocentric actions, which learns deep feature representations without relying on any pre-labeled action data. By combining convolutional and LSTM autoencoders, their method effectively discovers semantically meaningful action categories from raw video, outperforming supervised state-of-the-art methods on several challenging datasets.

What Problem Does This Paper Address?

Egocentric videos, captured from a first-person perspective (e.g., by wearable cameras like GoPro or Google Glass), are rapidly accumulating. Analyzing these videos is crucial for applications ranging from life logging to assistive vision. However, automatically recognizing actions in egocentric videos is particularly challenging due to several factors:

  • Annotation Difficulty: Manually labeling egocentric videos is time-consuming and expensive, especially given the sheer volume of data. Privacy concerns also make annotation more complex.
  • Video Characteristics: Egocentric videos are often long, unconstrained, and suffer from significant camera shake and motion blur due to head movements, making them difficult for traditional computer vision algorithms.
  • Action Diversity: Egocentric actions can range from simple object interactions (e.g., "picking up a cup") to complex activities (e.g., "driving") and can be short-term or long-term. Existing methods often require hand-tuned features specific to certain action types, limiting their generalization.

This paper addresses the need for a robust, generic, and unsupervised method that can learn to cluster egocentric actions without requiring any prior knowledge of action labels.

Methods

The authors propose a two-stage unsupervised approach:

  1. Frame-Level Feature Representation:
    • They employ an ensemble of "weak learners" in the form of multiple small convolutional autoencoders. This is inspired by boosting techniques, where combining multiple simple models can yield better performance than a single complex one.
  2. Each autoencoder is trained on a subset of video "splices" (short, consecutive video segments) to learn a specific representation.
  3. Inputs to these autoencoders are dense optical flow (for motion) and raw RGB frames (for appearance). The authors chose these modalities for their generality.
  4. The learned features from these frame-level autoencoders are then concatenated to form a rich representation for each frame.
  5. Temporal Feature Learning and Clustering:
    • To capture the temporal dynamics of actions, they use pooled frame-level features as input to multiple LSTM (Long Short-Term Memory) autoencoder networks.
  6. Each LSTM autoencoder is designed to capture temporal information at different resolutions, forming a "temporal pyramid." This allows the model to learn patterns at various time scales, crucial for both short-term and long-term actions.
  7. The temporal representations from these LSTM networks are concatenated.
  8. Finally, K-Means clustering is applied to these aggregated temporal features to group similar video splices into semantically meaningful action clusters.

The entire process is unsupervised, meaning it only requires raw video input without any explicit action labels during training.

Datasets

The authors evaluated their approach on four public egocentric action datasets:

  • GTEA: A dataset for short-term, hand-object coordinated actions.
  • ADL-short: Another dataset for short-term, hand-object coordinated actions.
  • ADL-long: A dataset for long-term, hand-object coordinated actions.
  • HUJIEGOSEG: A dataset for long-term actions without explicit hand-object interactions.

They also briefly experimented on the 50 SALAD dataset (a third-person action dataset) to demonstrate the generality of their approach.

Key Insights and Ideas

  • Ensemble of Weak Learners: Using multiple small autoencoders instead of one large one is more manageable for training and allows for learning diverse representations.
  • Temporal Pyramid: Capturing temporal information at multiple resolutions (via stacked LSTMs) is critical for understanding the full spectrum of egocentric actions, from quick gestures to extended activities.
  • Unsupervised Generality: The proposed method demonstrates that deep features learned in an unsupervised manner can be highly effective for clustering actions, even outperforming supervised methods that have access to labels. This is particularly important given the annotation challenges of egocentric data.
  • Semantic Clusters: The discovered clusters are shown to correspond to semantically meaningful actions, suggesting that the learned features capture the essence of human activities.
  • Input Modality Complementarity: Combining optical flow (motion) and RGB frames (appearance) provides complementary information, leading to better performance, especially for actions involving hand-object interactions.

Key Results

  • The proposed unsupervised approach achieved state-of-the-art accuracies on all four egocentric action datasets, surpassing supervised methods without using any action labels.
  • For example, on the GTEA dataset, their method achieved an accuracy of 0.76 (using flow+frame), compared to supervised methods that achieved 0.68. On HUJIEGOSEG, they achieved 0.90, outperforming the supervised state-of-the-art of 0.89.
  • The method successfully discovered semantically meaningful action clusters, which could be mapped to ground truth labels with reasonable accuracy (e.g., F1 scores around 0.5-0.6 for different levels of granularity).
  • The approach is robust to the choice of clustering assignment algorithm (e.g., greedy vs. Hungarian).

Limitations and Future Work

  • Background Class: The authors note that the "background" class is a significant limitation. Since it encompasses all segments not explicitly labeled as an action, it becomes a highly diverse and difficult class to learn a consistent representation for.
  • Temporal Regularization: While the method doesn't explicitly enforce temporal smoothness in cluster assignments, the results show that adjacent splices often fall into the same cluster, indicating some inherent temporal coherence. However, explicit temporal regularization could potentially further improve results.
  • Hyperparameter Sensitivity: The number of clusters in K-Means is an important hyperparameter. The paper shows that varying this allows for clustering at different semantic granularities, but finding the optimal number for a given task might require further investigation.
  • Future Work: The authors suggest that their approach could be extended to third-person action recognition and that further exploration of different input modalities and more sophisticated clustering techniques could yield even better results. They also highlight the potential for discovering finer-grained action details by increasing the number of clusters.

Summary updated

BibTeX
@inproceedings{bhatnagar2017unsupervised,
  author = {Bharat Lal Bhatnagar and Suriya Singh and Chetan Arora and C. V. Jawahar},
  title = {{Unsupervised Learning of Deep Feature Representation for Clustering Egocentric Actions}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/200},
}
Figure