Learning from Label Relationships in Human Affect

Niki Maria Foteinopoulou, Ioannis Patras

TL;DR

This paper tackles the challenge of accurately estimating human affect and mental state from long, in-the-wild videos, which often suffer from poor label resolution and limited data. They introduce a novel "relational loss" that leverages relationships between labels to improve generalization and a two-stage attention architecture that incorporates temporal context from neighboring video clips.

What Problem Does This Paper Address?

Estimating human affect (emotions, mood) and mental state severity from videos is crucial for applications in healthcare, education, and entertainment. However, this task is fraught with difficulties:

  • Poor Temporal Resolution: Labels often describe the entire video, not specific moments, making it hard to pinpoint when an emotion or symptom occurs.
  • Limited Data: Datasets are frequently small, partly due to privacy concerns, leading to overfitting in deep learning models.
  • Long Videos: "In-the-wild" videos can be very long, posing computational challenges and requiring models to understand long-range temporal dependencies.

These issues result in deep learning models that struggle to generalize well to unseen data, producing latent representations that are not robust.

Methods

The authors propose a two-pronged approach:

  1. Relational Regression Loss ($L_{rel}$): This is the core novelty. Instead of just minimizing the difference between predicted and ground truth labels, this loss encourages the model to learn latent representations where the distances between labels in the ground truth space are mirrored by the distances between their corresponding feature representations in the latent space. Essentially, if two video clips have similar labels, their learned features should also be similar. This is achieved by comparing cosine similarity matrices of both label vectors and feature vectors within a mini-batch.
  2. Two-Stage Attention Architecture:
    • Video-Clip Encoder: A standard ResNet backbone extracts frame-level features, which are then fed into a Transformer encoder to capture temporal relationships within a clip, producing clip-level features.
  3. Context-Based Attention: This module takes the clip-level features of the current clip and features from neighboring clips (context clips) to enrich the representation. It uses an attention mechanism where the current clip's features act as queries to attend to the context features (keys and values). This allows the model to incorporate temporal context beyond the immediate clip.

The overall framework jointly trains these components, optimizing both a standard regression loss ($L_{reg}$) and the proposed relational loss ($L_{rel}$).

Datasets

The proposed method was evaluated on three datasets:

  • NESS: A dataset for schizophrenia symptom severity estimation, featuring long, in-the-wild clinical interviews.
  • OMG-Emotion: A dataset for continuous affect estimation (Arousal/Valence) from in-the-wild videos.
  • AMIGOS: Another dataset for continuous affect estimation, also featuring in-the-wild videos.

The authors did not introduce new benchmarks but focused on demonstrating the effectiveness of their method on existing ones.

Key Insights and Ideas

  • Leveraging Label Structure: The most significant insight is that the relationships between labels (e.g., how similar or dissimilar different emotions or symptoms are) contain valuable information that can regularize learning and improve generalization, especially in low-data regimes.
  • Context is King (for Affect): Incorporating temporal context from neighboring clips significantly aids in understanding subtle affective states, which often unfold over time and are influenced by preceding events.
  • End-to-End Context Integration: Unlike prior methods that might pre-compute context features, this approach integrates context dynamically within an end-to-end trainable architecture, leading to faster training and better feature optimization.
  • Bridging Affect and Mental Health: The paper highlights the methodological parallels between continuous affect estimation and mental health symptom assessment, allowing for a unified approach and evaluation across these domains.

Key Results

  • Schizophrenia Severity: The proposed method significantly outperforms previous state-of-the-art methods on the NESS dataset, achieving a Pearson's Correlation Coefficient (PCC) of up to 78%, which is close to human expert performance (85%) and a substantial uplift of up to 40% over prior works.
  • Affect Recognition (OMG & AMIGOS): The method also excels on affect recognition tasks, outperforming previous vision-based methods in terms of Concordance Correlation Coefficient (CCC). Specifically, on AMIGOS, it achieved a 9% and 13% improvement for Arousal and Valence, respectively. On OMG, it showed up to a 5% improvement.
  • Ablation Studies: The relational loss ($L_{rel}$) consistently improved performance across datasets, demonstrating its effectiveness. Incorporating context features also boosted results, particularly for Valence. Unsupervised contrastive pre-training, while explored, showed less benefit compared to the proposed relational loss, likely due to the small dataset sizes.

Limitations and Future Work

  • Dataset Size: The paper acknowledges that while their method improves generalization without large datasets, performance could likely be further enhanced with more data. The effectiveness of unsupervised pre-training was limited, possibly due to the need for massive datasets for such approaches.
  • Computational Complexity: While the Transformer-based approach is noted as more efficient than RNNs for shorter sequences, the overall architecture still involves significant computation.
  • Future Work: The authors suggest exploring more sophisticated attention mechanisms and further investigating the interplay between different modalities (e.g., audio, text) for a more comprehensive understanding of human affect and mental state. The potential for applying the relational loss to other multi-label regression tasks is also implied.

Summary updated

Abstract

Human affect and mental state estimation in an automated manner, face a number of difficulties, including learning from labels with poor or no temporal resolution, learning from few datasets with little data (often due to confidentiality constraints) and, (very) long, in-the-wild videos. For these reasons, deep learning methodologies tend to overfit, that is, arrive at latent representations with poor generalisation performance on the final regression task. To overcome this, in this work, we introduce two complementary contributions. First, we introduce a novel relational loss for multilabel regression and ordinal problems that regularises learning and leads to better generalisation. The proposed loss uses label vector inter-relational information to learn better latent representations by aligning batch label distances to the distances in the latent feature space. Second, we utilise a two-stage attention architecture that estimates a target for each clip by using features from the neighbouring clips as temporal context. We evaluate the proposed methodology on both continuous affect and schizophrenia severity estimation problems, as there are methodological and contextual parallels between the two. Experimental results demonstrate that the proposed methodology outperforms all baselines. In the domain of schizophrenia, the proposed methodology outperforms previous state-of-the-art by a large margin, achieving a PCC of up to 78% performance close to that of human experts (85%) and much higher than previous works (uplift of up to 40%). In the case of affect recognition, we outperform previous vision-based methods in terms of CCC on both the OMG and the AMIGOS datasets. Specifically for AMIGOS, we outperform previous SoTA CCC for both arousal and valence by 9% and 13% respectively, and in the OMG dataset we outperform previous vision works by up to 5% for both arousal and valence.
BibTeX
@inproceedings{foteinopoulou2022learning,
  author = {Niki Maria Foteinopoulou and Ioannis Patras},
  title = {{Learning from Label Relationships in Human Affect}},
  booktitle = {ACM Multimedia},
  year = {2022},
  doi = {10.1145/3503161.3548373},
}
Figure