TL;DR
This paper introduces a novel architecture for recognizing social impressions (warmth and competence) by modeling the dynamic "causal" relationship between a speaker's behavior and a listener's reaction. By treating the speaker and listener as distinct but related data domains and using a combination of causality gating and cross-domain attention, the researchers achieved state-of-the-art results on the Dyadic IMPRESSION dataset.
What Problem Does This Paper Address?
In social psychology, "impressions" aren't just about what a speaker does; they are formed in the space between two people. While we often think of impressions like warmth or competence as inherent traits, they are actually perceptions filtered through a listener.
Most existing AI models for impression recognition fall into one of two traps: they either look only at the speaker’s behavior (ignoring the listener’s reaction) or they treat the speaker and listener as a single, unified data source. The authors argue that there is a "perception gap"—a disconnect between the signals a speaker sends and how a listener interprets them. To bridge this gap, a model needs to understand the causality of the interaction (how the listener reacts to specific speaker cues) and the domain difference (the fact that speaker and listener signals are fundamentally different types of information).
Methods
The authors propose a Listener Adaptive Cross-Domain Architecture consisting of two main stages:
- Listener Adaptation (LA):
- Causality Modeling: Instead of assuming all speaker behavior is equally important, the model uses Segmental PWCCA (Projection-Weighted Canonical Correlation Analysis). It splits the interaction into segments and calculates how strongly a listener’s physiological or behavioral response correlates with the preceding speaker segment. This "causality gating" weights the speaker's features based on how much they actually influenced the listener.
- Listener Modeling: Since different people have different biases when rating others, the model incorporates unique "Listener IDs" via one-hot encoding to account for individual annotation styles.
- Cross-Domain Fusion:
- Attention Mechanisms: The model uses Intra-attention (finding salient features within the speaker or listener's own signals) and Inter-attention (finding relevant connections between the two).
- Regularization Losses: To ensure the model doesn't just "overfit" to one side, they use two specific losses:
- Knowledge Distillation (KD) Loss: A "soft" way to transfer information between domains using Mean Squared Error.
- Similarity Enhancement (SE) Loss: Uses KL-divergence to align the representations, ensuring the model recognizes that the listener's state is a direct reflection of the speaker's influence.
Datasets
The study utilizes the Dyadic IMPRESSION dataset.
- Scope: 31 dyads (pairs), totaling 1,890 minutes of synchronized recordings.
- Modality: Includes audio, facial video, eye gaze, and peripheral physiological signals (EEG, ECG, blood volume pulse).
- Labels: Continuous, real-time annotations for "Warmth" and "Competence."
- Context: 40 listeners watched 13 different "stimuli" speakers, making it a robust benchmark for how different people perceive the same set of behaviors.
Key Insights and Ideas
- Causality is King: The most significant finding was that modeling the causal link—the idea that a listener's smile or heart rate spike is a reaction to a specific speaker cue—is more important than just throwing all the data into a fusion layer.
- The Perception Gap as a Feature, Not a Bug: By treating the speaker and listener as two different "domains" (like translating between two languages), the model can better capture the nuance of social perception.
- Soft vs. Hard Relatedness: The authors distinguish between "hard relatedness" (direct attention) and "soft relatedness" (knowledge distillation). They found that allowing the domains to "absorb" information from each other while maintaining independence leads to better generalization.
Key Results
The model was evaluated using the Concordance Correlation Coefficient (CCC), which measures how well the predicted trend matches the ground truth.
- Competence: Achieved 78.8%, a significant jump from the previous state-of-the-art of 77.0%.
- Warmth: Achieved 77.5%, outperforming the previous best of 75.1%.
- Ablation Study: Removing the "Causality Modeling" component caused the largest drop in performance (1.5% for competence and 1.9% for warmth), proving that the timing and reaction-based weighting are critical for understanding human impressions.
Limitations and Future Work
While the results are impressive, the authors note that the current model is tailored to the specific dynamics of the Dyadic IMPRESSION dataset. They suggest that future work should focus on:
- Generalization: Testing the architecture on other dyadic interaction scenarios (e.g., job interviews or counseling) to see if the causality weights hold up across different social contexts.
- Real-time Interaction: The current dataset involves listeners watching videos; applying this to live, two-way conversations where both parties are simultaneously speakers and listeners would be the next major hurdle.
Summary updated