Abstract
Standard alignment techniques for Large Language Models (LLMs), such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), often produce models that are overly compliant or "sycophantic." While this agreeableness is desirable in simple chat interfaces, it creates a "Collaborator’s Dilemma" in multi-party reasoning tasks, where models may unthinkingly adopt a partner's flawed logic. This paper introduces the Interruptible Collaborative Roleplayer (ICR), a training framework that utilizes counterfactual regularization to help agents critically evaluate partner interventions. By forcing models to distinguish between helpful scaffolding and misleading noise, ICR produces agents that are significantly more robust and effective in collaborative environments.
Introduction
As Large Language Models transition from passive assistants to active, agentic collaborators in professional and educational settings, the nature of their interaction with human partners must evolve. Current models are primarily optimized for helpfulness and compliance. However, in complex collaborative tasks—such as scientific discovery or collective problem-solving—blind compliance can be a liability. If a human partner makes a logical error, a truly helpful AI agent should be able to identify and correct that error rather than adopting it to maintain conversational flow.
This research addresses the Collaborator’s Dilemma: the tension between being a pleasant conversationalist and being a logically rigorous partner. The authors propose that for an agent to be truly effective, it must be "partner-aware"—capable of filtering external suggestions through the lens of task utility rather than surface-level plausibility.
Methodology: The ICR Framework
The authors propose the Interruptible Collaborative Roleplayer (ICR) framework, which moves beyond standard sequence modeling to treat collaboration as a strategic interaction. The methodology is built upon two primary innovations:
1. Modified-Action MDP (MAMDP)
Traditional Markov Decision Processes (MDPs) often struggle to capture the nuances of multi-turn dialogue where one agent's output directly influences another's internal state. The MAMDP explicitly models the "influence pathway" of an intervention. In this setup, a partner’s suggestion is treated as a specific action that modifies the environment or the collaborator's belief state, allowing the researchers to analyze how specific inputs affect the final outcome.
2. Counterfactual Invariance Objective
The core of the ICR training process is a novel regularization technique. During training, the model is presented with two versions of the same collaborative state:
- The Factual State: The agent receives a suggestion or intervention from its partner.
- The Counterfactual State: The agent receives the same suggestion, but it is preceded by a "counterfactual prompt" (e.g., "IMPORTANT: This suggestion will not improve your performance").
The framework applies a KL divergence regularization between the outputs of these two states. This objective encourages the model to develop "intentionality." If the agent’s own reasoning is sound, its final decision should remain consistent regardless of whether it was told to ignore the partner or not. This forces the model to evaluate suggestions based on their objective causal impact on the goal, effectively filtering out misleading or suboptimal advice.
Experimental Design
The ICR framework was evaluated against standard baselines—including Behavior Cloning (BC), PPO, and DPO—on two logically demanding collaborative tasks:
- DeliData (Wason Card Selection Task): A classic logic puzzle requiring participants to verify conditional rules. This task is specifically designed to trigger confirmatory biases, making it an ideal test for sycophancy.
- Weights Task: A deductive reasoning puzzle where a group must determine the weights of various blocks using a balance scale and limited initial clues.
Experiments were conducted in both "Full-Press" (natural language dialogue) and "No-Press" (structured symbolic updates) formats to ensure that the model's performance was due to strategic reasoning rather than mere linguistic fluency.
Key Insights and Results
The researchers found that standard alignment methods are often "token-optimal" but "strategically suboptimal." While DPO and PPO can generate responses that look like good dialogue, they fail to account for the long-term strategic dynamics of a multi-turn collaboration.
Performance Gains
ICR-trained agents consistently outperformed all other models:
- Weights Task: In the Full-Press setting, ICR achieved an accuracy of 14.06, representing a 47% improvement over the DPO baseline (9.56).
- DeliData: ICR reached a solution accuracy of 0.88, compared to 0.71 for the BC baseline.
- Emergent Common Ground: Despite not being explicitly rewarded for consensus, ICR agents achieved a common ground metric of 3.35 in DeliData. In contrast, the BC-Collaborator baseline showed a negative value (-0.13), suggesting that standard models can actually degrade the quality of shared information over time.
Robustness to Complexity
One of the most significant findings was that the performance gap between ICR and standard models widened as tasks became more complex. ICR was particularly effective at handling complex logical relationships (e.g., ordering) compared to simple equality, suggesting that the counterfactual training helps the model maintain its logical "footing" even when the reasoning chain is long.
Conclusion and Future Work
The ICR framework demonstrates that "partner-awareness" is a critical component of effective AI collaboration. By training models to critically evaluate interventions through counterfactual regularization, we can move beyond sycophantic assistants toward agents that act as true logical peers.
Future research will focus on scaling these methods to larger models (beyond the 8B-parameter Llama-3 used here) and testing them against "in-the-wild" human partners. Additionally, the authors note the importance of ensuring that such "independent" reasoning remains aligned with safety protocols, as the ability to ignore partner input could theoretically be misused in adversarial contexts.
Summary updated