TL;DR
This paper introduces a novel framework to evaluate how well Large Language Models (LLMs) understand and respond to social signals embedded in text, using Appraisal Theory. Their experiments reveal that current LLMs are more sensitive to positive social signals and struggle with negative ones, offering a crucial step towards building more socially intelligent AI.
What Problem Does This Paper Address?
As LLMs become increasingly integrated into our daily lives, their ability to navigate complex social interactions is paramount. A key challenge lies in their capacity to process and respond appropriately to "social signals" – subtle cues in language that convey emotions, judgments, attitudes, and relationship dynamics. While LLMs can generate human-like text, it's unclear how well they truly grasp the nuances of human social communication. This paper aims to bridge this gap by systematically evaluating LLM sensitivity to specific social signals, grounded in Appraisal Theory, which breaks down how language expresses emotions, judgments, and appreciation.
Methods
The researchers developed an experimental framework using a vignette study design. They crafted prompts that included a persona, a scenario, and an utterance embedded with specific social signals from Appraisal Theory: Affect (emotional state), Judgment (ethics/morals), and Appreciation (value of practices). These signals were presented with either positive or negative polarity. LLMs were then asked to perform two tasks:
- Analysis (Insight): Explain the linguistic framing of the social signals in the utterance compared to a neutral baseline.
- Response: Generate a conversational reply that aligns with the persona and responds to the social signals.
They used GPT-3.5 and GPT-4 for their experiments. To quantify the LLM outputs, they employed:
- Topical Modeling (for Insight): Using the Empath tool to analyze the presence of specific themes related to the social signals.
- Social Intelligence Metrics (for Response): Measuring emotional polarity, emotional intensity, and empathy.
Datasets
The study did not introduce new datasets for training. Instead, it utilized:
- Appraisal Theory Framework: As the theoretical basis for defining and categorizing social signals.
- Empathic Conversations dataset: Used for fine-tuning models to measure social intelligence dimensions (emotional polarity, intensity, and empathy) in the LLM responses.
- "Persuasion for Good" scenario: Adapted as the conversational context for the experiments.
Key Insights and Ideas
- Positive Bias: LLMs show a clear preference for responding to positive social signals, demonstrating higher sensitivity to them compared to negative ones.
- Struggle with Negativity: LLMs exhibit limited sensitivity to negative social signals, particularly in the realms of judgment and appreciation. This suggests a potential blind spot in their social understanding.
- Persona Impact on Response, Not Insight: While personas influenced the LLMs' conversational responses, they did not significantly alter the analytical "Insight" generated about the social signals.
- "Analyze then Respond" Boosts Engagement: Requiring LLMs to first analyze the social signals before responding led to more engaged and persona-aligned replies.
Key Results
- Statistically Significant Sensitivity: Both GPT-3.5 and GPT-4 demonstrated statistically significant sensitivity to all three social signals (Affect, Judgment, Appreciation) in their generated insights.
- Affect's Strongest Influence: Affect had the most pronounced impact on the "Optimism" Empath category for insights.
- Response to Negative Affect: Negative Affect significantly increased empathy and emotional polarity in responses, though the intensity varied between GPT-3.5 and GPT-4.
- Limited Sensitivity to Negative Evaluations: LLMs showed increased empathy and emotional intensity for positive judgments and appreciation, but limited sensitivity to negative social evaluations.
- Persona Consistency: Response patterns were largely consistent across different personas, though some personas (like the leader) showed lower empathy scores.
Limitations and Future Work
- Focus on GPT Models: The study was limited to GPT-3.5 and GPT-4. Future work should explore a wider range of LLMs to see if these sensitivity patterns are consistent.
- Selective Output Measures: The study used specific measures (Empath categories, social intelligence metrics). Expanding the range of output measures could provide a more comprehensive view of LLM capabilities.
- Broader Social Signals: The framework could be extended to include a wider array of social signals beyond Appraisal Theory.
- Diverse Scenarios: Evaluating LLMs in more varied social scenarios and with different language framing would offer deeper insights.
Summary updated