SEMA: Semantic Attention for Capturing Long-Range Dependencies in Egocentric Lifelogs

Pravin Nagar, K.n. Ajay Shastry, Jayesh Chaudhari, Chetan AroraWACV 2024

TL;DR

This paper introduces SEMA (Semantic Attention), a novel attention mechanism designed to efficiently capture long-range dependencies in egocentric video lifelogs. By factorizing the self-attention matrix using semantically meaningful representative frames, SEMA overcomes the quadratic complexity limitations of standard transformers, enabling effective activity clustering in extremely long, unsupervised video sequences.

What Problem Does This Paper Address?

The paper tackles the challenge of understanding activity patterns in egocentric lifelogs, which are characterized by extremely long sequences (spanning days or even weeks) and a lack of annotated data due to privacy concerns. Traditional transformer architectures, while powerful for modeling global dependencies, suffer from quadratic space and time complexity with respect to sequence length, making them intractable for such long videos. Existing methods to address this, like sparse attention or low-rank approximations (e.g., NMF-based approaches), struggle to capture the rich spatio-temporal visual cues scattered across these lengthy, often repetitive, lifelogs. The goal is to develop a representation learning framework that can effectively cluster activities from these long, unlabeled egocentric videos.

Methods

The core contribution is the SEMA (Semantic Attention) mechanism. It builds upon the idea of factorizing the self-attention matrix into low-rank query (Q) and key (K) matrices, inspired by NMF. However, instead of using random projections (like in Performer) or predefined kernels, SEMA leverages semantically meaningful representative frames.

Here's how it works:

  • Representative Frame Sampling: A set of representative frames is first extracted from the video sequence using a video summarization technique. These frames are chosen to capture the essence of different activities.
  • Semantic Factorization: The self-attention matrix is then factorized by projecting the query and key matrices onto a subspace defined by these representative frames. This is mathematically expressed as $A = Q'K'$, where $Q' = \exp(QRT)$ and $K' = \exp(RKT)$, with $R$ being the matrix of representative frame features. This approach aims to focus attention on semantically relevant parts of the sequence.
  • Self-Supervised Clustering Pipeline: To train the representation learning network (SEMAFormer), a self-supervised approach is employed. The process iteratively:
  • Learns embeddings using the SEMAFormer.
  • Generates pseudo-labels by clustering these embeddings using a core-set algorithm.
  • Updates the SEMAFormer using these pseudo-labels via a cross-entropy loss. This EM-like iterative process allows the model to learn cluster-centric representations without explicit ground truth labels.

  • Spectral Clustering: Finally, spectral clustering is applied to the learned embeddings to detect activity patterns.

Datasets

The paper uses several datasets for evaluation:

  • EgoRoutine: A publicly available dataset comprising lifelogs from seven subjects over 104 days, captured by a wearable camera. This dataset is particularly relevant due to its length and unconstrained nature. The authors also provide annotations for this dataset for their experiments.
  • UTE: A dataset used to demonstrate generalization. It consists of subjects performing daily activities for up to 5 hours. The authors also annotated these videos.
  • Epic Kitchens: A synthesized long video sequence (approx. 20k frames) was created from this dataset for evaluation.
  • THUMOS14 and AVA: Used for evaluating SEMA's application to standard video analysis tasks like online action detection and video recognition.
  • ActivityNet 1.3: Used for action localization tasks.

Key Insights and Ideas

  • Semantic Focus is Crucial: The key insight is that simply approximating self-attention with low-rank factorization isn't enough for long, complex sequences like lifelogs. Focusing on semantically meaningful representative frames is critical for capturing relevant spatio-temporal cues.
  • Representative Frames as Semantic Anchors: Using representative frames acts as a form of "semantic anchoring," guiding the attention mechanism to relevant parts of the sequence, rather than relying on arbitrary spatial or random projections.
  • Self-Supervision for Privacy-Sensitive Data: The proposed self-supervised clustering pipeline is vital for egocentric data, where ground truth labels are often unavailable due to privacy concerns. The iterative pseudo-labeling and representation learning effectively bypasses the need for manual annotation.
  • SEMA's Generalizability: The paper demonstrates that SEMA is not just for activity clustering but can also be effectively substituted for standard self-attention in other video analysis tasks, often with significant memory savings.

Key Results

  • Significant Improvement in Activity Clustering: SEMA achieves significant performance gains (e.g., 8-15% improvement in NMI, AMI, and F-Score on EgoRoutine) compared to state-of-the-art approaches for egocentric lifelog activity clustering.
  • Efficiency Gains: SEMA offers substantial reductions in GPU memory consumption compared to standard self-attention, making it more practical for long sequences and potentially for edge deployments. For example, in action detection, SEMA uses 813MB of GPU memory compared to 1195MB for standard self-attention with a batch size of 1.
  • Competitive Performance on Standard Tasks: When applied to online action detection, video recognition, and action localization, SEMA achieves performance on par with standard self-attention mechanisms, while still maintaining lower memory footprints.
  • Robustness to Skewed Distributions: The self-supervised approach with SEMA is shown to be robust to the highly skewed activity distributions often found in egocentric lifelogs, outperforming methods that rely on equipartition assumptions.

Limitations and Future Work

  • Dependency on Video Summarization: The performance of SEMA is partly dependent on the quality of the representative frames extracted by the video summarization technique.
  • Misclassifications due to Overlap: Qualitative results show that SEMA can still misclassify activities when there is high visual overlap in context or objects (e.g., "food in lab" vs. "walking in lab and chitchatting").
  • Future Work: The authors suggest that SEMA's semantic factorization approach is a generic idea that can be extended to other video analysis tasks requiring long-range contextual understanding. Further exploration of different video summarization techniques and more sophisticated ways to handle overlapping activities could be future directions.

Summary updated

BibTeX
@inproceedings{nagar2024sema,
  author = {Pravin Nagar and K.N. Ajay Shastry and Jayesh Chaudhari and Chetan Arora},
  title = {{SEMA: Semantic Attention for Capturing Long-Range Dependencies in Egocentric Lifelogs}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure