Introduction
In human-computer interaction, "addressee recognition"—the ability to identify who is being spoken to—is a fundamental challenge for developing socially intelligent agents. While humans perform this task instinctively, AI systems frequently struggle in "in-the-wild" scenarios such as crowded public spaces or social gatherings.
Most prior research has been confined to controlled laboratory settings or specific tasks, often relying solely on non-verbal cues like head pose or eye gaze. However, gaze is an imperfect signal; for instance, if two people are looking at a shared object while talking, the speaker is not looking at the addressee. This research argues that to truly understand social context, an AI must integrate visual scene information with the textual content of the utterance.
Methodology
The authors proposed an end-to-end multi-modal deep learning architecture designed to process diverse social signals. The model consists of three primary processing streams:
- Saliency-Estimation Stream: Utilizes a pre-trained VGGNet on the original image to understand the surrounding environment and identify potential objects or people drawing the speaker's attention.
- Speaker-Appearance Stream: Employs a second VGGNet to process a cropped image of the speaker along with their annotated head location, capturing posture and orientation.
- Utterance-Based Stream: Uses a Long Short-Term Memory (LSTM) network to process the text of the utterance. Words are mapped into a geometric space using GloVe (Global Vectors for Word Representation) embeddings.
Features from these streams are concatenated and passed through a fully connected layer with a Softmax function to classify the addressee into three categories: "Line-of-Sight Entities," "Photographer" (the AI's perspective), or "Others" (including monologues).
The ARVSU Dataset
To evaluate the model, the researchers introduced the ARVSU (Addressee Recognition in Visual Scenes with Utterances) benchmark. Built upon the GazeFollow dataset, which contains over 122,000 images, the authors used Amazon Mechanical Turk to collect nearly 700,000 simulated utterances. Crowdsourced workers provided likely sentences and identified intended addressees for specific individuals in the images. Unlike previous task-specific datasets, ARVSU provides a vast variety of real-life conversational scenarios.
Key Insights and Results
The evaluation revealed several critical findings regarding how different modalities contribute to social understanding:
- The Power of Text: Surprisingly, the text-only model (60.7% accuracy) significantly outperformed the vision-only model (54.0%). This suggests that the semantic content of speech often carries more weight in determining the addressee than visual orientation alone.
- Multi-modal Synergy: Combining vision and text yielded the highest overall accuracy at 62.5%, proving that these modalities provide complementary information.
- Gaze Disagreement: The study found that in nearly 10% of cases where a person addresses the "photographer" (the AI), they are not looking at the camera. This explains why gaze-only systems often fail in natural interactions.
- Classification Performance: The model was most effective at recognizing "Line-of-Sight Entities" (71.4% F1-score) but found the "Photographer" class more challenging (40.1% F1-score) due to the aforementioned gaze disagreement and the difficulty of distinguishing monologues from direct address.
Limitations and Future Directions
While the multi-modal approach shows promise, several challenges remain. The current model relies on annotated head locations; future iterations will need to integrate automatic detection algorithms for full autonomy. Additionally, the researchers plan to incorporate physical distance cues, hypothesizing that proximity can help distinguish between a speaker addressing the AI and a person engaged in a monologue. Finally, addressing potential biases in crowdsourced simulated speech remains a priority for improving real-world robustness.
Summary updated