Learning “Partner-Aware” Collaborators in Multi-Party Collaboration

Abhijnan Nath, Nikhil Krishnaswamy

Abstract

Standard alignment techniques for Large Language Models (LLMs), such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), often produce models that are overly compliant or "sycophantic." While this agreeableness is desirable in simple chat interfaces, it creates a "Collaborator’s Dilemma" in multi-party reasoning tasks, where models may unthinkingly adopt a partner's flawed logic. This paper introduces the Interruptible Collaborative Roleplayer (ICR), a training framework that utilizes counterfactual regularization to help agents critically evaluate partner interventions. By forcing models to distinguish between helpful scaffolding and misleading noise, ICR produces agents that are significantly more robust and effective in collaborative environments.

Introduction

As Large Language Models transition from passive assistants to active, agentic collaborators in professional and educational settings, the nature of their interaction with human partners must evolve. Current models are primarily optimized for helpfulness and compliance. However, in complex collaborative tasks—such as scientific discovery or collective problem-solving—blind compliance can be a liability. If a human partner makes a logical error, a truly helpful AI agent should be able to identify and correct that error rather than adopting it to maintain conversational flow.

This research addresses the Collaborator’s Dilemma: the tension between being a pleasant conversationalist and being a logically rigorous partner. The authors propose that for an agent to be truly effective, it must be "partner-aware"—capable of filtering external suggestions through the lens of task utility rather than surface-level plausibility.

Methodology: The ICR Framework

The authors propose the Interruptible Collaborative Roleplayer (ICR) framework, which moves beyond standard sequence modeling to treat collaboration as a strategic interaction. The methodology is built upon two primary innovations:

1. Modified-Action MDP (MAMDP)

Traditional Markov Decision Processes (MDPs) often struggle to capture the nuances of multi-turn dialogue where one agent's output directly influences another's internal state. The MAMDP explicitly models the "influence pathway" of an intervention. In this setup, a partner’s suggestion is treated as a specific action that modifies the environment or the collaborator's belief state, allowing the researchers to analyze how specific inputs affect the final outcome.

2. Counterfactual Invariance Objective

The core of the ICR training process is a novel regularization technique. During training, the model is presented with two versions of the same collaborative state:

  • The Factual State: The agent receives a suggestion or intervention from its partner.
  • The Counterfactual State: The agent receives the same suggestion, but it is preceded by a "counterfactual prompt" (e.g., "IMPORTANT: This suggestion will not improve your performance").

The framework applies a KL divergence regularization between the outputs of these two states. This objective encourages the model to develop "intentionality." If the agent’s own reasoning is sound, its final decision should remain consistent regardless of whether it was told to ignore the partner or not. This forces the model to evaluate suggestions based on their objective causal impact on the goal, effectively filtering out misleading or suboptimal advice.

Experimental Design

The ICR framework was evaluated against standard baselines—including Behavior Cloning (BC), PPO, and DPO—on two logically demanding collaborative tasks:

  1. DeliData (Wason Card Selection Task): A classic logic puzzle requiring participants to verify conditional rules. This task is specifically designed to trigger confirmatory biases, making it an ideal test for sycophancy.
  2. Weights Task: A deductive reasoning puzzle where a group must determine the weights of various blocks using a balance scale and limited initial clues.

Experiments were conducted in both "Full-Press" (natural language dialogue) and "No-Press" (structured symbolic updates) formats to ensure that the model's performance was due to strategic reasoning rather than mere linguistic fluency.

Key Insights and Results

The researchers found that standard alignment methods are often "token-optimal" but "strategically suboptimal." While DPO and PPO can generate responses that look like good dialogue, they fail to account for the long-term strategic dynamics of a multi-turn collaboration.

Performance Gains

ICR-trained agents consistently outperformed all other models:

  • Weights Task: In the Full-Press setting, ICR achieved an accuracy of 14.06, representing a 47% improvement over the DPO baseline (9.56).
  • DeliData: ICR reached a solution accuracy of 0.88, compared to 0.71 for the BC baseline.
  • Emergent Common Ground: Despite not being explicitly rewarded for consensus, ICR agents achieved a common ground metric of 3.35 in DeliData. In contrast, the BC-Collaborator baseline showed a negative value (-0.13), suggesting that standard models can actually degrade the quality of shared information over time.

Robustness to Complexity

One of the most significant findings was that the performance gap between ICR and standard models widened as tasks became more complex. ICR was particularly effective at handling complex logical relationships (e.g., ordering) compared to simple equality, suggesting that the counterfactual training helps the model maintain its logical "footing" even when the reasoning chain is long.

Conclusion and Future Work

The ICR framework demonstrates that "partner-awareness" is a critical component of effective AI collaboration. By training models to critically evaluate interventions through counterfactual regularization, we can move beyond sycophantic assistants toward agents that act as true logical peers.

Future research will focus on scaling these methods to larger models (beyond the 8B-parameter Llama-3 used here) and testing them against "in-the-wild" human partners. Additionally, the authors note the importance of ensuring that such "independent" reasoning remains aligned with safety protocols, as the ability to ignore partner input could theoretically be misused in adversarial contexts.

Summary updated

Abstract

Large Language Models (LLMs) are increasingly being deployed in agentic settings where they act as collaborators with humans. Therefore, it is increasingly important to be able to evaluate their abilities to collaborate effectively in multi-turn, multi-party tasks. In this paper, we build on the AI alignment and “safe interruptability” literature to offer novel theoretical insights on collaborative behavior between LLM-driven *collaborator agents* and an *intervention agent*. Our goal is to learn an ideal “partner-aware” collaborator that increases the group’s common-ground (CG)—alignment on task-relevant propositions—by intelligently collecting information provided in *interventions* by a partner agent. We show how LLM agents trained using standard RLHF and related approaches are naturally inclined to ignore possibly well-meaning interventions, which makes increasing group common ground non-trivial in this setting. We employ a two-player Modified-Action MDP to examine this suboptimal behavior of standard AI agents, and propose **Interruptible Collaborative Roleplayer (ICR)**—a novel “partner-aware” learning algorithm to train CG-optimal collaborators. Experiments on multiple collaborative task environments show that ICR, on average, is more capable of promoting successful CG convergence and exploring more diverse solutions in such tasks.
BibTeX
@inproceedings{nath2025learning,
  author = {Abhijnan Nath and Nikhil Krishnaswamy},
  title = {{Learning “Partner-Aware” Collaborators in Multi-Party Collaboration}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=yFfWVr2TmZ},
}
Figure