D-vlog: Multimodal Vlog Dataset for Depression Detection

Jeewoo Yoon, Chaewon Kang, Seungbae Kim, Jinyoung HanAAAI 2022 · DOI

Abstract

Traditional depression detection relies on clinical interviews, which are often reactive and limited by laboratory settings. This paper introduces D-Vlog, a naturalistic multimodal dataset of 961 YouTube vlogs (~160 hours) designed to capture behavioral indicators of depression "in the wild." We propose a deep learning architecture utilizing a cross-attention mechanism to fuse acoustic and visual features. Our results demonstrate that models trained on D-Vlog generalize better to clinical datasets than vice versa, with acoustic features proving particularly predictive.

Introduction

Early screening for depression is critical for timely intervention. However, automated detection systems face challenges due to the scarcity of realistic data. Existing datasets like DAIC-WOZ are typically recorded in controlled environments, which may not reflect how individuals express symptoms in daily life. D-Vlog addresses this by leveraging public YouTube vlogs where individuals voluntarily share personal experiences, providing a diverse and naturalistic source of behavioral data.

The D-Vlog Dataset

D-Vlog consists of 961 videos from 816 distinct speakers, collected between January 2020 and January 2021. The dataset was manually annotated by trained researchers who identified explicit mentions of depressive symptoms or clinical status.

Cumulative distribution of vlog duration and frequency per channel.

As shown in the distributions above, the dataset captures a wide range of vlog lengths and frequencies, providing a robust basis for longitudinal behavioral analysis. Compared to traditional datasets, D-Vlog offers a significantly larger number of subjects, enhancing the model's ability to generalize across different individuals.

Methodology

Our proposed Depression Detector pipeline involves three primary stages: feature extraction, unimodal encoding, and multimodal fusion.

Feature Extraction

We extract features from two modalities:

  1. Acoustic Features: Using the OpenSmile toolkit, we extract 25 low-level descriptors (LLDs) including loudness, pitch (MFCCs), and spectral flux.
  2. Visual Features: Using the dlib library, we extract 68 facial landmarks (represented as 136-dimensional vectors) to capture micro-expressions and facial movements.
Acoustic and visual feature extraction pipelines.

Model Architecture

Each modality is processed through a Transformer encoder to capture temporal dependencies. To fuse these streams, we employ a cross-attention mechanism. This allows the model to learn correlations between modalities—for instance, how specific vocal inflections align with facial expressions. The fused representation is then passed through a final Transformer layer and a softmax function for classification.

Results and Discussion

The Depression Detector achieved an F1-score of 63.50%, outperforming standard baselines like Bi-LSTM (59.70%) and Tensor Fusion Networks (61.00%).

Key Findings

  • Modality Importance: Audio-only models (58.55% F1) outperformed visual-only models (56.38%), suggesting that vocal patterns like slowed speech and reduced intensity are more consistent indicators of depression in vlogs.
  • Generalization: Models trained on D-Vlog performed well when tested on clinical datasets (DAIC-WOZ), indicating that "in the wild" data captures a more robust set of features than laboratory-based data.
  • Gender Disparity: The model showed higher accuracy for male subjects, suggesting that depressive indicators in vlogs may manifest more distinctly in men or that current features better capture male behavioral changes.

Conclusion and Future Work

D-Vlog provides a valuable resource for developing non-invasive, early screening tools for depression. Future research will explore the integration of textual features (NLP) and address cultural variations in emotional expression. Additionally, mitigating demographic biases remains a priority to ensure equitable performance across all user groups.

Summary updated

BibTeX
@inproceedings{yoon2022d,
  author = {Jeewoo Yoon and Chaewon Kang and Seungbae Kim and Jinyoung Han},
  title = {{D-vlog: Multimodal Vlog Dataset for Depression Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21483},
}
Figure