Mental-Perceiver: Audio-Textual Multi-Modal Learning for Estimating Mental Disorders

Jinghui Qin, Changsong Liu, Tianchi Tang, Dahuang Liu, Minghao Wang, Qianying Huang, Rumin ZhangAAAI 2025 · DOI

Introduction

Early detection of mental disorders like anxiety and depression is critical, yet clinical interviews are often expensive, time-consuming, and prone to concealment by patients—especially children and adolescents. While AI-driven screening offers a scalable alternative, the field is currently limited by small, adult-centric, and primarily English-language datasets. To address these gaps, researchers have introduced MMPsy, the first large-scale Mandarin-speaking adolescent dataset for mental health detection, and Mental-Perceiver, a novel deep learning architecture designed to fuse audio and textual data effectively.

Methodology: The Mental-Perceiver Architecture

The core innovation of the Mental-Perceiver lies in its use of "category semantic priors"—pre-calculated anchors representing "normal" versus "disordered" textual states. These priors act as prototypes that guide the model during training.

Architecture of the Mental-Perceiver model featuring category semantic priors and cross-attention mechanisms.

The architecture follows a transformer-based approach inspired by PerceiverIO:

  1. Category Semantic Priors: Average text representations for each category are derived from the training set to serve as discriminative guides.
  2. Cross-Attention Mechanism: High-dimensional multimodal inputs (audio Mel-spectrograms and BERT-encoded text) are mapped into a fixed-size latent space using the semantic priors as queries.
  3. Deep Feature Extraction: Eight layers of self-attention refine the latent representations.
  4. Dual Training Objectives: The model is optimized via a standard classification loss and a matching loss that aligns input data with the appropriate category prior.

Datasets: MMPsy and DAIC-WOZ

The study introduces MMPsy (Multi-Modal Psychological assessment corpus), a benchmark featuring Mandarin-speaking adolescents. After rigorous cleaning, the dataset includes 7,736 samples for anxiety and 4,247 for depression, with labels derived from GAD-7 and PHQ-9 questionnaires. To test generalizability, the researchers also evaluated the model on DAIC-WOZ, a standard English-language clinical interview dataset.

Key Results and Insights

Mental-Perceiver consistently outperformed strong baselines, including XGBoost, ConvLSTM, and large-scale models like Qwen2-Audio-Instruct, across different languages and contexts.

Performance comparison of Mental-Perceiver against baseline methods on MMPsy and DAIC-WOZ datasets.

Key findings include:

  • The Power of Priors: Providing "prototypes" of mental states stabilizes training and helps the model navigate high-dimensional noise in raw audio.
  • Multimodal Synergy: Ablation studies confirmed that while text is a strong predictor, adding audio features (tone, prosody, hesitation) significantly boosts Unweighted Average Recall (UAR), highlighting that how something is said is as vital as what is said.
  • Cross-Dataset Robustness: Achieving 0.79 accuracy on the English DAIC-WOZ dataset demonstrates that the model learns robust, cross-linguistic features of psychological distress.

Limitations and Future Directions

While promising, the MMPsy dataset is currently focused on students in Guangdong, China. Future research should investigate if these patterns hold across different age groups and regional dialects. Additionally, incorporating video data (facial expressions) and advanced self-supervised speech models like Wav2Vec 2.0 could further enhance detection accuracy while maintaining privacy standards.

Summary updated

BibTeX
@inproceedings{qin2025mental,
  author = {Jinghui Qin and Changsong Liu and Tianchi Tang and Dahuang Liu and Minghao Wang and Qianying Huang and Rumin Zhang},
  title = {{Mental-Perceiver: Audio-Textual Multi-Modal Learning for Estimating Mental Disorders}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i23.34687},
}
Figure