TL;DR
The AVEC 2017 challenge aimed to advance the automatic detection of depression and emotions from real-world audiovisual data. This paper details the novelties of the 2017 iteration, including the introduction of "likability" as an affect dimension and a continuous estimation of depression severity, alongside the challenge guidelines, datasets, and baseline system performance.
What Problem Does This Paper Address?
Accurately recognizing human emotions and mental states like depression from natural, unscripted interactions is a significant challenge. Existing methods often rely on controlled lab environments or limited datasets, which don't reflect the complexities of real-life scenarios. The AVEC (Audio/Visual Emotion Challenge and Workshop) series aims to bridge this gap by providing standardized benchmarks and datasets for researchers to compare their approaches. AVEC 2017 specifically tackled the limitations of previous challenges by focusing on "in-the-wild" data and introducing new tasks: predicting "likability" alongside arousal and valence, and estimating depression severity as a continuous value rather than a binary classification. This addresses the need for more robust and generalizable affect and depression recognition systems applicable to real-world human-computer and human-human interactions.
Methods
AVEC 2017 featured two sub-challenges: Affect Sub-Challenge (ASC) and Depression Sub-Challenge (DSC).
For the ASC, participants were tasked with continuously predicting arousal, valence, and a new dimension, "likability." The primary evaluation metric was the Concordance Correlation Coefficient (CCC), which accounts for both correlation and agreement between predicted and ground truth time series. Novelties included the use of the SEWA dataset, recorded "in-the-wild," and the inclusion of likability.
For the DSC, participants estimated depression severity as a continuous value using the DAIC-WOZ dataset. The primary metric was Root Mean Square Error (RMSE). This refined the previous year's binary classification task to a more nuanced continuous estimation.
Both sub-challenges allowed participants to use their own feature extraction methods but also provided standard feature sets for audio, video, and text. Baseline systems for both challenges utilized Support Vector Regression (SVR) for emotion recognition and Random Forest Regression for depression estimation. A key methodological aspect was the use of time-shifted features to account for potential delays in real-time annotation and prediction.
Datasets
- Affect Sub-Challenge (ASC): A subset of the SEWA (Sentiment Analysis in the Wild) database was used. This dataset consists of "in-the-wild" audiovisual recordings of dyadic conversations, specifically focusing on German subjects discussing a commercial.
- Depression Sub-Challenge (DSC): The DAIC-WOZ (Distress Analysis Interview Corpus - Wizard of Oz) database was utilized. This corpus contains clinical interviews collected as part of an effort to create a computer agent for identifying mental illness indicators.
Key Insights and Ideas
- "In-the-wild" data is crucial: The use of datasets like SEWA, recorded in natural environments, presents a more realistic and challenging benchmark for affect recognition compared to lab-controlled settings.
- Continuous depression estimation is more informative: Moving from binary depression classification to continuous severity estimation provides a richer and more nuanced understanding of a person's mental state.
- Likability as a new dimension: Introducing "likability" as a target for affect recognition opens new avenues for understanding social interactions and user experience.
- Feature fusion remains important: Baseline results indicated that combining audio, video, and text modalities generally led to improved performance in both emotion and depression recognition tasks.
- Time-shifting features helps: Accounting for reaction time delays in real-time annotation and prediction is a necessary step for improving model performance.
Key Results
- ASC Baseline Performance: The baseline system achieved CCC scores on the development partition ranging from 0.081 (liking, audio) to 0.507 (valence, multimodal). On the test partition, the best unimodal performance was for video (0.455 for valence), while multimodal fusion yielded the highest CCC for arousal (0.466).
- DSC Baseline Performance: The baseline random forest regression model achieved an RMSE of 6.62 and MAE of 5.52 on the development set for the audio-video fusion. On the test set, the audio-video fusion achieved an RMSE of 7.05 and MAE of 5.66. A simple baseline of predicting the mean score resulted in an RMSE of 5.73.
Limitations and Future Work
- Data limitations: While "in-the-wild" data was used, the SEWA dataset for ASC was a subset, and the DAIC-WOZ dataset for DSC, while rich, still represents specific interview scenarios.
- Baseline system performance: The baseline results, while competitive, highlight the ongoing challenge of achieving highly accurate continuous prediction, especially for certain modalities and dimensions (e.g., liking in audio).
- Future work suggested by the challenge: The paper implies that future work should focus on improving robustness to real-world variations, exploring more sophisticated fusion techniques, and potentially developing models that can better capture subtle emotional and depressive cues. The availability of baseline scripts is intended to improve reproducibility and facilitate future research.
Summary updated