Measuring the Unspoken: A Disentanglement Model and Benchmark for Psychological Analysis in the Wild

Yigui Feng, Qinglin Wang, Haotian Mo, Yang Liu, Ke Liu, Gencheng Liu, Xinhai Chen, Siqi Shen, Songzhu Mei, Jie LiuarXiv 2025 · DOI · arXiv

Abstract

Vision-Language Models (VLMs) often fail to distinguish between the physical mechanics of speech and genuine emotional expressions, a phenomenon termed "Articulatory-Affective Ambiguity." This paper presents MIND (Multilevel Insight Network for Disentanglement), a novel framework that disentangles speech-related facial movements from emotional signals. By leveraging a hierarchical visual encoder and a specialized status judgment module, MIND achieves state-of-the-art performance in psychological analysis. The authors also introduce ConvoInsight-DB, a high-resolution dataset with expert psychological annotations, and PRISM, an LLM-based evaluation metric for generative psychological reasoning.

The Challenge: Articulatory-Affective Ambiguity

Creating AI that genuinely understands human psychology in natural, unconstrained conversations is incredibly difficult. Current multimodal research faces two primary bottlenecks:

  1. Articulatory-Affective Ambiguity: The same facial muscles used to express emotion are used to speak. Current VLMs aggregate visual features holistically, making them architecturally blind to the difference. They frequently misinterpret the visual patterns of articulation (e.g., opening the mouth wide to pronounce a vowel) as affective signals (e.g., a gasp of shock).
  2. The Evaluation Void: Standard text generation metrics like BLEU or ROUGE are insufficient for evaluating complex, generative psychological analysis. There is no standardized way to measure why a model infers a specific emotional state based on visual grounding.

The MIND Framework

To address these issues, the authors propose the Multilevel Insight Network for Disentanglement (MIND). This hierarchical visual encoder is designed to filter out speech-related visual noise to isolate genuine emotional cues.

The MIND architecture features a Status Judgment Module to disentangle speech from emotion and a hierarchical encoding path for micro-expressions.

Core Technical Contributions

  • Spatiotemporal Feature Decomposition: MIND extracts four distinct facial dynamics streams: head pose, eye dynamics, core emotion, and lip movement.
  • Status Judgment Module: This module applies a temporal contrastive test to the lip movement stream. If variance crosses a threshold (indicating speech), the model algorithmically suppresses lip features, preventing the downstream LLM from being misled by articulation mechanics.
  • Hierarchical Encoding: Features are processed through two parallel paths: a transient path for fleeting micro-expressions and a fusion path for complex dynamics.

Benchmarking and Evaluation

Alongside the model, the researchers introduced ConvoInsight-DB, a large-scale dataset crawled from movies and TV shows. Unlike previous datasets with coarse labels, ConvoInsight-DB features expert annotations covering macro-expressions, micro-expressions, and deep psychological inference.

To evaluate generative outputs, they developed PRISM (Psychological Reasoning Insight Scoring Metric). Using GPT-4o as an expert judge, PRISM scores model outputs across four dimensions: macro-expressions, micro-expressions, psychological insight, and detail coverage.

Key Results and Performance

The MIND-LLM-8B model significantly outperformed both general-purpose and specialized emotion models. A standout result was the +86.95% performance improvement in micro-expression detection compared to the best prior baseline.

A qualitative comparison showing MIND-LLM's superior ability to detect micro-expressions and provide deep psychological reasoning compared to other emotional LMMs.

Ablation studies confirmed that the Status Judgment module is critical; its removal led to a 16.86% drop in micro-expression detection, proving that filtering speech noise is a fundamental prerequisite for accurate psychological analysis.

Limitations and Ethical Considerations

While MIND represents a major leap forward, it currently relies on visual heuristics for speech detection, which may fail with "subtle talkers" or expressive non-speakers. Furthermore, the model is currently "deaf," relying entirely on vision without integrating audio prosody.

The authors emphasize that this technology should not be used as a lie detector or diagnostic tool without rigorous auditing, citing significant risks regarding surveillance, privacy, and cultural bias.

Summary updated

Abstract

Generative psychological analysis of in-the-wild conversations faces two fundamental challenges: (1) existing Vision-Language Models (VLMs) fail to resolve Articulatory-Affective Ambiguity, where visual patterns of speech mimic emotional expressions; and (2) progress is stifled by a lack of verifiable evaluation metrics capable of assessing visual grounding and reasoning depth. We propose a complete ecosystem to address these twin challenges. First, we introduce Multilevel Insight Network for Disentanglement(MIND), a novel hierarchical visual encoder that introduces a Status Judgment module to algorithmically suppress ambiguous lip features based on their temporal feature variance, achieving explicit visual disentanglement. Second, we construct ConvoInsight-DB, a new large-scale dataset with expert annotations for micro-expressions and deep psychological inference. Third, Third, we designed the Mental Reasoning Insight Rating Metric (PRISM), an automated dimensional framework that uses expert-guided LLM to measure the multidimensional performance of large mental vision models. On our PRISM benchmark, MIND significantly outperforms all baselines, achieving a +86.95% gain in micro-expression detection over prior SOTA. Ablation studies confirm that our Status Judgment disentanglement module is the most critical component for this performance leap. Our code has been opened.
BibTeX
@misc{feng2025measuring,
  author = {Yigui Feng and Qinglin Wang and Haotian Mo and Yang Liu and Ke Liu and Gencheng Liu and Xinhai Chen and Siqi Shen and Songzhu Mei and Jie Liu},
  title = {{Measuring the Unspoken: A Disentanglement Model and Benchmark for Psychological Analysis in the Wild}},
  year = {2025},
  eprint = {2512.04728},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.04728},
}
Figure