Introduction
In current AI pipelines, understanding human communication for psychological assessment is often a fragmented, two-step process. Typically, a speech-to-text model like Whisper transcribes audio, and a separate text-based Language Model (LM) analyzes that transcript for meaning or psychological traits. This approach is computationally redundant and suffers from a "representation gap," where text-based models consistently outperform audio models on human-level tasks like personality or emotion detection.
WhiSPA (Whisper with Semantic and Psychological Alignment) bridges this gap by training the audio model to internalize high-performing text-based representations from the start, creating a single, efficient audio encoder capable of sophisticated psychological analysis.
Methodology: Student-Teacher Distillation
The researchers employed a student-teacher learning paradigm to align audio and text embeddings into a unified cross-modal space.
As illustrated in the architecture above, the framework consists of:
- The Student: A pre-trained Whisper model (tested in "tiny" and "medium" sizes).
- The Teacher: A combination of SBERT (Sentence-BERT) for general semantics and PsychEmb, a set of lexically derived embeddings covering ten psychological dimensions, including the Big Five personality traits, Valence, Arousal, and symptoms of Depression or Anxiety.
- Alignment Variants: The study compared WhiSA (aligning only with SBERT's semantic space) and WhiSPA (aligning with both SBERT and the explicit psychological dimensions of PsychEmb).
A critical component of this alignment is the Noise Contrastive Estimation (NCE) loss function. Unlike simple cosine similarity, NCE actively pushes unrelated audio-text pairs apart while pulling similar ones together, creating a more discriminative and robust latent space.
Bridging the Modality Gap
Before alignment, speech and text embeddings exist in "different neighborhoods" of the latent space. WhiSPA effectively bridges this gap, creating a shared space where an audio clip and its transcript reside in nearly the same location.
This alignment allows the model to capture necessary semantic depth directly from audio, rendering the traditional secondary text-LM pipeline optional. Furthermore, the model's interpretability is enhanced; analysis shows that WhiSPA's predictions correlate with meaningful linguistic markers, such as associating symptom-related words with higher distress scores, mirroring the diagnostic patterns of human clinicians.
Experimental Results
The study utilized two large-scale clinical datasets: WTC-Segments (250+ hours from World Trade Center responders) and HiTOP-Segments (470+ hours of clinical interviews).
Key performance metrics include:
- Massive Error Reduction: Compared to the baseline Whisper model, WhiSPA achieved an average error reduction of 73.4% on self-supervised tasks and 83.8% on downstream psychological tasks.
- Scaling Performance: When scaled to a Whisper-medium backbone, error reduction reached 86.61%.
- Teacher Parity: In many scenarios, WhiSPA's audio-only representations performed as well as, or even better than, the text-based SBERT teacher.
- State-of-the-Art Benchmarks: WhiSPA set new benchmarks for person-level psychological assessments, such as predicting "Internalizing" or "Disinhibition" traits directly from raw speech.
Limitations and Future Directions
The primary limitation of the current framework is lexical bias. Because the teacher model is text-based, WhiSPA learns to focus heavily on what is being said (the words) rather than how it is being said (prosody, tone, and pitch).
Future work will focus on developing dual loss objectives to preserve raw acoustic features while maintaining semantic alignment. Additionally, the researchers plan to test the framework on general-purpose tasks like Emotion Recognition in Conversation (ERC) and explore deployment in low-latency mental health screening tools.
Summary updated