Nunchi: Ears Help Eyes in Multi-Party Conversations

JunYeong Park, Shijing Wang, Seonwook Park, Michael Stengel, Shalini De Mello, Sang-Il Choi, Hyung Jin ChangarXiv 2026 · arXiv

In social settings, we don't just use our eyes to figure out who someone is looking at; we use our ears, too. We listen for who is speaking and what they are saying to "read the room." This social intuition is known in Korean as Nunchi.

A new paper titled "Nunchi: Ears Help Eyes in Multi-Party Conversations" introduces a framework that brings this human-like reasoning to computer vision, demonstrating that conversational context is the key to solving gaze ambiguity in crowded scenes.

TL;DR

Nunchi is a framework that identifies which person an observer is looking at by combining traditional gaze geometry with conversational cues (who is speaking and what is said). By feeding these cues into a Vision-Language Model (VLM), Nunchi outperforms existing gaze-only models by nearly 5 percentage points, with even larger gains in crowded scenes where geometry alone is ambiguous.

What Problem Does This Paper Address?

Identifying the "attended person" is crucial for human-robot interaction and social analysis. However, existing models rely almost exclusively on gaze geometry—the vector from the observer's eyes to a target.

As shown in the figure below, geometry becomes highly ambiguous as more people enter a scene. If multiple candidates are standing in a similar direction from the observer, a tiny error in gaze estimation leads the model to pick the wrong person. While humans naturally use the flow of conversation to resolve this (e.g., we usually look at the person currently speaking), AI models have lacked both the datasets and the architectures to integrate these "ears" with their "eyes."

Figure 1. Attended person identification in a multi-party con- versation. As more people appear in a scene, several candidates lie in a similar direction from the observer, so gaze geometry alone cannot determine whom the observer looks at. Gaze-LLE [35] and MTGS [15] each select an incorrect candidate. Nunchi combines gaze geometry with conversational cues, the speaker and the tran- script, and identifies the attended person C2.

Methods

The authors propose Nunchi, a three-stage framework that treats attended person identification as a reasoning task for a Vision-Language Model (VLM).

  1. Cue Extraction: Instead of training a massive end-to-end model, Nunchi uses "frozen experts." It employs Gaze-LLE to generate a gaze heatmap and LR-ASD (an active speaker detector) to score how likely each person is to be speaking. It also pulls in the transcript of the conversation.
  2. Prompt Construction: The core innovation is converting these heterogeneous cues into a unified text prompt. The VLM receives the original image (with candidates labeled $C_1, C_2, ... C_k$), a crop of the observer, and a text description of the expert outputs (e.g., "Gaze-LLE score for $C_1$ is 0.514," "Speaker cue: $C_2$ is speaking").
  3. Candidate Scoring: A VLM (specifically Qwen3-VL-8B) reads the prompt and the images. It is fine-tuned to output the label of the attended person. Because all cues are presented in a shared language space, the VLM can explicitly weigh a weak gaze signal against a strong conversational cue.
Figure 3. Overview of Nunchi. Cue extraction obtains gaze and speaker information with frozen experts, along with the transcript. Prompt construction writes them as text, and every cue refers to each candidate by the label Ck drawn in the keyframe. Candidate scoring reads the two images and the prompt with a single VLM and scores each candidate by the logit of its label. In this example, the gaze prior and the speaker cue point to different candidates, and Nunchi selects the attended person C2. Colors mark the gaze prior, speaker, and transcript paths, and ✓/✗mark agreement with the ground truth, which the model does not see.

Datasets

The researchers found that existing gaze datasets lacked the synchronized audio and transcripts needed for this task. To fill this gap, they introduced SORBET (SOcial gaze Recordings with Both spEech and Transcripts).

  • SORBET: Contains 3,340 samples from 1,578 clips across three TV series (Friends, The Big Bang Theory, and Superstore). Unlike previous datasets, every sample includes synchronized video, audio, speaker labels, and transcripts, specifically focusing on multi-party scenes with 3+ people.
  • Evaluation: The model was also tested on existing datasets like VGS and AVA-LAEO to prove its generalizability.
Figure 2. Construction of SORBET. From two multi-party conversation datasets, we keep frames with at least three identified people and a visible speaker, and take the middle frame of each segment of such frames as a keyframe. Each identified person serves in turn as the observer, and the annotator labels what the observer looks at. SORBET keeps samples whose gaze target is a candidate.

Key Insights and Ideas

  • Conversational Cues > Visual Cues: Surprisingly, adding more visual information like depth maps or mutual gaze indicators did not improve performance. However, adding the speaker cue provided a significant boost.
  • Transcripts Need Context: The transcript cue (what is said) is most helpful once the model already knows who is speaking. For example, if a transcript says "Ross, come here," the model can infer the observer is looking at the speaker only if it knows which candidate is currently talking.
  • VLMs as Reasoning Engines: Rather than using complex fusion modules to mix feature vectors, the authors found that simply "writing down" the expert findings in a prompt allowed the VLM to perform sophisticated multi-modal reasoning.

Key Results

  • SOTA Performance: Nunchi achieved 86.7% accuracy on SORBET, outperforming the strongest gaze-only baseline (Gaze-LLE) by 4.8 percentage points.
  • Scaling with Crowd Size: The advantage of Nunchi grows with the number of people. In scenes with 5 or more candidates, Nunchi outperformed the best baseline by 8.0 percentage points.
  • Robustness: The gains held steady even when using automatic transcripts (Whisper) instead of human-written ones, losing only 0.3 points in accuracy.

Limitations and Future Work

The authors acknowledge several limitations:

  • Scripted Content: SORBET is built from TV shows, where acting and camera framing might differ from real-world, spontaneous interactions.
  • Static Keyframes: The current model identifies the target based on a single keyframe. Future work could incorporate temporal information (video) to handle gaze shifts and more complex social dynamics.
  • Identity Dependency: The system currently relies on knowing character identities to link names in transcripts to faces in the image. Generalizing this to "stranger" conversations remains a challenge.

Nunchi demonstrates that for AI to truly understand social interaction, it needs to do more than just look—it needs to listen.

Summary updated

Abstract

Identifying whom a person looks at is central to understanding multi-party conversations, yet gaze geometry alone becomes ambiguous as more people appear. Conversations offer cues that geometry lacks, such as who is speaking and what is said, but no gaze dataset provides them for many multi-party scenes. We introduce SORBET, a benchmark of 3,340 samples from 1,578 clips with synchronized speech, transcripts, and attended person labels, covering more multi-party scenes than existing gaze datasets. Prior gaze models struggle to use these cues, which come in different forms, must be linked to the right person, and can conflict with gaze geometry. We propose Nunchi, a framework that constructs a text prompt from the transcript and the outputs of gaze and speaker models, links every cue to the candidates with shared labels, and lets a vision-language model (VLM) weigh gaze geometry against conversational cues to identify the attended person. Nunchi outperforms the strongest existing method by 4.8 percentage points. On SORBET, conversational cues help whereas additional visual cues do not, and the gains hold across three VLMs, with automatic transcripts, and on two existing datasets.
BibTeX
@misc{park2026nunchi,
  author = {JunYeong Park and Shijing Wang and Seonwook Park and Michael Stengel and Shalini De Mello and Sang-Il Choi and Hyung Jin Chang},
  title = {{Nunchi: Ears Help Eyes in Multi-Party Conversations}},
  year = {2026},
  eprint = {2611.23115},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23115},
}
Figure