In social settings, we don't just use our eyes to figure out who someone is looking at; we use our ears, too. We listen for who is speaking and what they are saying to "read the room." This social intuition is known in Korean as Nunchi.
A new paper titled "Nunchi: Ears Help Eyes in Multi-Party Conversations" introduces a framework that brings this human-like reasoning to computer vision, demonstrating that conversational context is the key to solving gaze ambiguity in crowded scenes.
TL;DR
Nunchi is a framework that identifies which person an observer is looking at by combining traditional gaze geometry with conversational cues (who is speaking and what is said). By feeding these cues into a Vision-Language Model (VLM), Nunchi outperforms existing gaze-only models by nearly 5 percentage points, with even larger gains in crowded scenes where geometry alone is ambiguous.
What Problem Does This Paper Address?
Identifying the "attended person" is crucial for human-robot interaction and social analysis. However, existing models rely almost exclusively on gaze geometry—the vector from the observer's eyes to a target.
As shown in the figure below, geometry becomes highly ambiguous as more people enter a scene. If multiple candidates are standing in a similar direction from the observer, a tiny error in gaze estimation leads the model to pick the wrong person. While humans naturally use the flow of conversation to resolve this (e.g., we usually look at the person currently speaking), AI models have lacked both the datasets and the architectures to integrate these "ears" with their "eyes."
Methods
The authors propose Nunchi, a three-stage framework that treats attended person identification as a reasoning task for a Vision-Language Model (VLM).
- Cue Extraction: Instead of training a massive end-to-end model, Nunchi uses "frozen experts." It employs Gaze-LLE to generate a gaze heatmap and LR-ASD (an active speaker detector) to score how likely each person is to be speaking. It also pulls in the transcript of the conversation.
- Prompt Construction: The core innovation is converting these heterogeneous cues into a unified text prompt. The VLM receives the original image (with candidates labeled $C_1, C_2, ... C_k$), a crop of the observer, and a text description of the expert outputs (e.g., "Gaze-LLE score for $C_1$ is 0.514," "Speaker cue: $C_2$ is speaking").
- Candidate Scoring: A VLM (specifically Qwen3-VL-8B) reads the prompt and the images. It is fine-tuned to output the label of the attended person. Because all cues are presented in a shared language space, the VLM can explicitly weigh a weak gaze signal against a strong conversational cue.
Datasets
The researchers found that existing gaze datasets lacked the synchronized audio and transcripts needed for this task. To fill this gap, they introduced SORBET (SOcial gaze Recordings with Both spEech and Transcripts).
- SORBET: Contains 3,340 samples from 1,578 clips across three TV series (Friends, The Big Bang Theory, and Superstore). Unlike previous datasets, every sample includes synchronized video, audio, speaker labels, and transcripts, specifically focusing on multi-party scenes with 3+ people.
- Evaluation: The model was also tested on existing datasets like VGS and AVA-LAEO to prove its generalizability.
Key Insights and Ideas
- Conversational Cues > Visual Cues: Surprisingly, adding more visual information like depth maps or mutual gaze indicators did not improve performance. However, adding the speaker cue provided a significant boost.
- Transcripts Need Context: The transcript cue (what is said) is most helpful once the model already knows who is speaking. For example, if a transcript says "Ross, come here," the model can infer the observer is looking at the speaker only if it knows which candidate is currently talking.
- VLMs as Reasoning Engines: Rather than using complex fusion modules to mix feature vectors, the authors found that simply "writing down" the expert findings in a prompt allowed the VLM to perform sophisticated multi-modal reasoning.
Key Results
- SOTA Performance: Nunchi achieved 86.7% accuracy on SORBET, outperforming the strongest gaze-only baseline (Gaze-LLE) by 4.8 percentage points.
- Scaling with Crowd Size: The advantage of Nunchi grows with the number of people. In scenes with 5 or more candidates, Nunchi outperformed the best baseline by 8.0 percentage points.
- Robustness: The gains held steady even when using automatic transcripts (Whisper) instead of human-written ones, losing only 0.3 points in accuracy.
Limitations and Future Work
The authors acknowledge several limitations:
- Scripted Content: SORBET is built from TV shows, where acting and camera framing might differ from real-world, spontaneous interactions.
- Static Keyframes: The current model identifies the target based on a single keyframe. Future work could incorporate temporal information (video) to handle gaze shifts and more complex social dynamics.
- Identity Dependency: The system currently relies on knowing character identities to link names in transcripts to faces in the image. Generalizing this to "stranger" conversations remains a challenge.
Nunchi demonstrates that for AI to truly understand social interaction, it needs to do more than just look—it needs to listen.
Summary updated