TL;DR
This paper introduces AGENTVISS, a new benchmark designed to evaluate the visual social intelligence of Multimodal Large Language Models (MLLMs) in multi-agent simulations. By testing models on scenarios where verbal statements conflict with visual cues, the researchers uncover a stark "perception-to-decision" bottleneck: while current models excel at acting out assigned roles, they struggle to use raw visual data to dynamically regulate interactions and manage social outcomes.
What Problem Does This Paper Address?
While multi-turn LLM agents have shown impressive capabilities in negotiating, persuading, and role-playing, existing social benchmarks are almost entirely text-based. In the real world, however, human communication relies heavily on non-verbal signals like facial expressions, posture, gaze, and subtle emotional shifts. This is particularly critical when a person's words contradict their actual emotional state—such as someone claiming "I'm fine" while slumping their shoulders and avoiding eye contact.
Without multimodal benchmarks, we cannot evaluate whether AI agents can truly "read the room." This paper addresses this gap by creating a framework to test if MLLMs can exploit visual social cues to make appropriate interactional decisions.
Methods
The authors built a multimodal social simulation environment using structured role profiles and a Belief-Desire-Intention (BDI) framework augmented with social risk (BDI-Risk).
To evaluate the agents, the benchmark splits tasks into two main categories across four dimensions:
- Role Enactment Tasks:
- Expression Task: Does the agent maintain its assigned information-management style (e.g., deception, honest signaling)?
- Characteristic Task: Does the agent adhere to its conflict-handling stance (e.g., competing, accommodating)?
- Interaction Management Tasks:
- Interaction Regulation Task (Int.-Reg.): Does the agent adapt dynamically to others' evolving verbal and non-verbal signals?
- Interaction Outcome Task (Int.-Out.): Does the agent successfully achieve its intended social outcome?
Crucially, the authors test the models under two distinct observation modes to isolate where failures occur:
- Verbalized-Vision (VV): The agent is first prompted to convert the images into textual descriptions of facial expressions and postures before reasoning.
- Direct-Vision (DV): The agent directly consumes raw images alongside the text history.
Datasets
The authors introduced the AGENTVISS dataset, which consists of:
- 240 scenarios extracted from the first season of the TV show Friends, grounded in actual video segments.
- 585 role instances and 2,340 role-task instances.
- Four distinct dialogue types: Persuasion, Deliberation, Information-seeking, and Eristic (conflict-driven).
Each scenario includes a fixed group image capturing the interpersonal layout, dynamic role portraits that update across turns, textual descriptions, and structured role profiles.
Key Insights and Ideas
- The Perception-to-Decision Bottleneck: MLLMs suffer from an integration bottleneck rather than a pure perception failure. When forced to verbalize what they see first (VV mode), models perform significantly better at regulating interactions than when they look at raw images directly (DV mode). Human checks showed that models like Claude-Sonnet-4.6 can perceive visual details near-perfectly, but fail to translate those raw visual inputs into social decisions in DV mode.
- Local Enactment vs. Global Management: Current MLLMs are excellent at "acting" like a character locally (averaging over 94% on role enactment), but they fall short when required to track other participants' states and steer the conversation toward a social goal (averaging only ~57% on interaction management).
- Social Complexity Matters: Interaction regulation becomes substantially harder as group sizes grow (tracking 4-5 characters vs. 2) and varies by dialogue structure. Information-seeking tasks proved the hardest for interaction regulation, likely due to the asymmetrical nature of question-answering roles.
Key Results
The researchers evaluated seven recent MLLMs (including Claude-Sonnet-4.6, GPT-5.4, and Qwen3.5 variants).
- Interaction Regulation Gap: There is a massive performance drop from VV to DV mode on the Interaction Regulation task. Across all models, the average score dropped from 49.60 (VV) to 31.21 (DV). Claude-Sonnet-4.6 saw the most dramatic drop, plummeting from 71.75 to 34.10.
- Vision-Dependent Outcomes: When evaluating the Interaction Outcome task, tasks categorized as "strongly vision-dependent" (requiring concrete visual cues to verify success) saw an average score drop of nearly 20 points compared to weakly vision-dependent tasks.
- Text Alone is Insufficient: Both VV and DV modes consistently outperformed a text-only baseline, proving that visual evidence provides critical social information that text context alone cannot capture.
Limitations and Future Work
The authors note several limitations:
- Source Homogeneity: The scenarios are derived entirely from a single Western sitcom (Friends), meaning cross-cultural social norms and professional settings are underrepresented.
- Static vs. Continuous Vision: The benchmark uses static group images and portraits at fixed turns rather than continuous video, missing temporal nuances like micro-expressions and real-time gaze shifts.
- LLM-as-Judge Biases: Evaluation relies on an ensemble of LLM judges, which may share inherent biases with the agents they are scoring.
Future work points toward extending social simulation to continuous video inputs and exploring architectures that better integrate raw visual features directly into social decision-making frameworks.
Summary updated