Abstract
We propose a method for modeling and learning turn-taking behaviors for accessing a shared resource. We model the individual behavior for each agent in an interaction and then use a multi-agent fusion model to generate a summary over the expected actions of the group to render the model independent of the number of agents. The individual behavior models are weighted finite state transducers (WFSTs) with weights dynamically updated during interactions, and the multi-agent fusion model is a logistic regression classifier. We test our models in a multi-agent tower-building environment, where a Q-learning agent learns to interact with rule-based agents. Our approach accurately models the underlying behavior patterns of the rule-based agents with accuracy ranging between 0.63 and 1.0 depending on the stochasticity of the other agent behaviors. In addition, we show using KL-divergence that the model accurately captures the distribution of next actions when interacting with both a single agent (KL-divergence < 0.1) and with multiple agents (KL-divergence < 0.37). Finally, we demonstrate that our behavior model can be used by a Q-learning agent to take turns in an interactive turn-taking environment.
Introduction
In human society, turn-taking is governed by complex social-cultural norms. While conversational agents can handle interruptions—which are often "recoverable"—many physical tasks, such as merging into traffic or collaborative construction, require strict turn-taking because simultaneous actions lead to catastrophic failure.
The challenge for AI in these domains is twofold: scalability and diversity. Most multi-agent models struggle to generalize when the number of participants changes, and agents often exhibit different "personalities" (e.g., passive, aggressive, or erratic). This paper introduces a hierarchical framework that learns when to act (the social norm) separately from what to do (the task logic), allowing an agent to cooperate effectively with an arbitrary number of diverse teammates.
Methods: A Hierarchical Framework
The proposed architecture consists of three tiers designed to decouple individual behavior modeling from group dynamics.
- Individual Behavior Models (WFSTs): Each agent’s history is tracked using a Weighted Finite State Transducer. This model predicts an agent's next move—pass, indicate intent, or act—based on their previous actions and the current task state.
- Multi-Agent Fusion Model: To ensure independence from the number of agents, a Multinomial Logistic Regression classifier "summarizes" the group by agent type (e.g., predicting the likely next actions of all "Aggressive" agents). This summary provides a fixed-size input regardless of group size.
- Turn-Taking Policy (Q-Learning): A reinforcement learning agent uses the fusion model's outputs to decide its next move, learning to wait for its turn to avoid project-collapsing collisions.
The task logic (the "how") is handled by a separate Deep Q-Network (DQN) trained to place bricks optimally, allowing the focus to remain on the timing of turns.
Experimental Setup
The model was evaluated in a custom Multi-Agent Tower-Building Environment. Agents collaboratively build a tower using bricks of various sizes. The tower collapses if two agents attempt to place a brick simultaneously or if a placement offsets the center of gravity. The learning agent was tested against three rule-based profiles:
- Passive: Waits for others to act.
- Aggressive: Seizes every opportunity to act.
- Stochastic: Exhibits erratic or random behavior.
Key Results
The WFST models demonstrated high precision in predicting the actions of rule-based agents. Accuracy reached 1.0 for passive agents and approximately 0.9 for aggressive agents, while dropping to ~0.77 for stochastic agents as expected.
Using KL-divergence to measure the difference between probability distributions, the authors showed the model accurately captured group intent. KL-divergence remained low (< 0.1 for single agents and < 0.37 for multiple agents), indicating a strong match between the model's predictions and actual behavior distributions. Furthermore, the Q-learning agent successfully learned turn-taking policies even when interacting with highly erratic stochastic agents—a scenario where simpler baseline models failed.
Discussion and Future Work
A key insight from this work is that decoupling social timing from task execution makes the model significantly more robust. Additionally, the "summary by type" approach provides a scalable solution for multi-agent coordination. Interestingly, the study found that the presence of "well-behaved" agents (passive or aggressive) helps regulate the chaos of stochastic agents, facilitating policy convergence for the learning agent.
Future work will address current limitations, such as the use of discrete "intention" signals. Real-world scenarios often involve continuous signals, such as vehicle speed or hand movements. Additionally, the researchers plan to extend the model to "hypothesize" agent types in real-time rather than assuming they are known a priori, and to test the framework in more complex continuous environments like robotics or driving simulators.
Summary updated