Abstract
As a sub-branch of affective computing, impression recognition, e.g., perception of speaker characteristics such as warmth or competence, is potentially a critical part of both human-human conversations and spoken dialogue systems. Most research has studied impressions only from the behaviors expressed by the speaker or the response from the listener, yet ignored their latent connection. In this paper, we perform impression recognition using a proposed listener adaptive cross-domain architecture, which consists of a listener adaptation function to model the causality between speaker and listener behaviors and a cross-domain fusion function to strengthen their connection. The experimental evaluation on the dyadic IMPRESSION dataset verified the efficacy of our method, producing concordance correlation coefficients of 78.8% and 77.5% in the competence and warmth dimensions, outperforming previous studies. The proposed method is expected to be generalized to similar dyadic interaction scenarios.
Introduction
In social psychology, impressions are not static traits inherent to a speaker; rather, they are formed dynamically through the interaction between a speaker and a listener. Traditional AI models often overlook this "perception gap"—the disconnect between the signals a speaker emits and how a listener interprets them. Most existing approaches either focus solely on the speaker's behavior or treat the dyad as a single, undifferentiated data source. This paper addresses these limitations by explicitly modeling the causality of the interaction and the domain differences between speaker and listener signals.
Methodology: Listener Adaptive Cross-Domain Architecture
The proposed architecture bridges the perception gap through two primary stages: Listener Adaptation and Cross-Domain Fusion.
1. Listener Adaptation (LA)
The LA stage focuses on the influence of the speaker on the listener.
- Causality Modeling: Utilizing Segmental Projection-Weighted Canonical Correlation Analysis (PWCCA), the model segments the interaction to calculate the correlation between a listener’s physiological or behavioral response and the preceding speaker segment. This "causality gating" ensures that speaker features are weighted based on their actual impact on the listener.
- Listener Modeling: To account for individual biases in social perception, unique "Listener IDs" are incorporated via one-hot encoding, allowing the model to adapt to different annotation styles.
2. Cross-Domain Fusion
This stage treats the speaker and listener as distinct data domains to better capture the nuance of social perception.
- Attention Mechanisms: The model employs Intra-attention to identify salient features within a single domain and Inter-attention to map relevant connections between the speaker and listener.
- Regularization Losses: Two specific losses are used to align the representations: Knowledge Distillation (KD) Loss for soft information transfer and Similarity Enhancement (SE) Loss (via KL-divergence) to ensure the listener's state is recognized as a reflection of the speaker's influence.
Experimental Results
The model was evaluated on the Dyadic IMPRESSION dataset, which includes 1,890 minutes of synchronized audio, facial video, eye gaze, and physiological signals (EEG, ECG, blood volume pulse) from 31 dyads. Performance was measured using the Concordance Correlation Coefficient (CCC).
As shown in Figure 4, the proposed method achieved state-of-the-art results, with 78.8% for Competence and 77.5% for Warmth. The ablation study highlights that Causality Modeling is the most critical component; its removal led to the most significant performance drops (1.5% for competence and 1.9% for warmth). This confirms that the temporal, reaction-based weighting of features is essential for accurate impression recognition.
Conclusion
By modeling the causal link between speaker cues and listener reactions, this research demonstrates that social impressions are best captured as a dynamic, cross-domain interaction. The success of the Listener Adaptive Cross-Domain Architecture suggests it could be generalized to other dyadic scenarios, such as job interviews or therapeutic counseling. Future work will explore the application of this model to live, two-way interactions where both participants act as speaker and listener simultaneously.
Summary updated