Abstract
In a setting where two AI agents embodied as animated humanoid avatars are engaged in a conversation with one human and each other, we see two challenges. One, determination by the AI agents about which one of them is being addressed. Two, determination by the AI agents if they may/could/should speak at the end of a turn. In this work we bring these two challenges together and explore the participation of AI agents in multi-party conversations. Particularly, we show two embodied AI shopkeeper agents who sell similar items aiming to get the business of a user by competing with each other on the price. In this scenario, we solve the first challenge by using headpose (estimated by deep learning techniques) to determine who the user is talking to. For the second challenge we use deontic logic to model rules of a negotiation conversation.
Introduction
Most conversational AI systems are designed for one-on-one interactions using explicit "wake-words" (e.g., Siri or Alexa). However, natural human conversation is often multi-party and fluid. In a crowded marketplace, for instance, users rely on eye contact and body language rather than repeating a name for every utterance. This research addresses the complexities of multi-party dialogue by introducing a system where two AI "shopkeeper" agents compete to sell items to a human user in a virtual environment.
The authors identify two primary hurdles for AI in these settings:
- Addressee Detection: Determining if the human is speaking to a specific agent, the other agent, or themselves.
- Proactive Turn-Taking: Enabling agents to know when it is socially acceptable or strategically necessary to interrupt or respond, even if they were not the primary addressee.
Methodology
The proposed system integrates robust computer vision with symbolic logic to manage complex social interactions.
Robust Headpose Estimation
Traditional facial landmark detection often fails in real-world settings with low lighting or unconventional camera angles. To overcome this, the authors employed a deep learning approach that combines a generative model with a probabilistic deep model. This allows the system to estimate a user's gaze even with low-resolution images or extreme head tilts. If the user's gaze overlaps with an agent's avatar during speech, that agent is identified as the addressee.
Deontic Logic and the Ravel Tool
To govern the social dynamics of the conversation, the researchers utilized deontic logic—a branch of logic dealing with obligations, permissions, and prohibitions. Using a tool called Ravel, they implemented four core rules:
- R1: The user is always permitted to speak.
- R2: Agents are prohibited from responding to their own utterances (preventing infinite loops).
- R3: If an agent is directly addressed, they are obligated to respond, and others are prohibited from interrupting.
- R4: Agents are permitted to respond to a "price pitch" to facilitate market competition.
Results and Discussion
The system acts as a Finite State Machine where every potential response from an AI agent is passed through the Ravel logic engine. If the logic determines a response is "prohibited," the system blocks the message before it reaches the user.
As illustrated in Table 1, the logic engine effectively manages a competitive sequence. When the user addresses Agent A1 (R1), A1 responds with a price (R3). An immediate attempt by Agent A2 to offer a lower price is blocked by R3 because A1 was the primary addressee and held the conversational obligation. However, later in the interaction, A2 is permitted to broadcast a competitive offer under R4. This demonstrates how logic-based filtering allows for proactive, competitive behavior without violating social norms or creating chaotic interruptions.
Conclusion and Future Work
This work provides a successful proof of concept for multi-agent competitive dialogue using a combination of vision-based addressee detection and logic-based turn-taking. While the current social rules are hand-crafted, future research could explore using machine learning to automatically discover or refine these rules through observation. Additionally, scaling the system to larger groups of agents and conducting formal user evaluations will be essential to assess the perceived naturalness and helpfulness of the interaction in practical applications.
Summary updated