Walking the Tightrope: Autonomous Disentangling Beneficial and Detrimental Drifts in Non-Stationary Custom-Tuning

Xiaoyu Yang, Jie Lu, En YuNeurIPS 2025 · OpenReview

Introduction

Customizing Multi-modal Large Language Models (MLLMs) for specialized domains such as clinical medicine often relies on Reinforcement Fine-Tuning (RFT). While effective, these models frequently encounter "non-stationary" environments characterized by sparse data, noise, and long-tail distributions. In these contexts, models are prone to detrimental concept drift—a phenomenon where the model’s Chain-of-Thought (CoT) reasoning gradually deviates from clinical reality. This paper introduces Counterfactual Preference Optimization (CPO), a framework designed to stabilize reasoning by disentangling beneficial learning from harmful drift through causal inference and counterfactual analysis.

The Challenge: Catastrophic Error Propagation

The researchers identify a critical failure mode in autoregressive generation: tiny, unpredictable shifts in token probabilities can snowball during the reasoning process. For instance, a model might correctly identify "lung opacity" but, due to internal drift, conclude with an incorrect diagnosis of pneumonia instead of atelectasis. This "catastrophic error propagation" makes MLLMs inherently risky for safety-critical applications like radiology, where the causal path from visual evidence to diagnosis must remain rigorous.

Methodology

To address these drifts, the authors bridge the gap between concept drift theory and causal inference, treating the model's reasoning as a stream that can be mathematically analyzed and corrected.

1. Causal Modeling and Do-Calculus

Using Structural Causal Graphs, the authors identify concept drift as a "confounder"—a hidden variable that distorts the relationship between the input image and the final prediction. By treating the Chain-of-Thought as a "mediator," they utilize do-calculus to isolate the direct effect of the image on the diagnosis, effectively stripping away the "backdoor" influence of drift.

2. The CXR-CounterFact (CCF) Dataset

A major contribution of this work is the creation of the CXR-CounterFact (CCF) dataset. To build it, the researchers developed a hierarchical concept graph covering 12 diseases and 53 attributes. They then employed a medical expert model (Med-PaLM) to generate over 320,416 counterfactual reasoning trajectories. These trajectories represent "what-if" scenarios (e.g., "What would the reasoning look like if this were not pneumonia?"), providing a rich source of negative constraints for training.

3. Counterfactual Preference Optimization (CPO)

CPO is the core algorithmic contribution. Unlike standard Direct Preference Optimization (DPO), which focuses primarily on aligning with positive examples, CPO utilizes a dual-preference objective. It trains the model to maximize the likelihood of correct human reasoning while explicitly minimizing similarity to the adversarially generated counterfactual paths. This approach effectively decouples helpful learning from the harmful drifts inherent in non-stationary data.

Experimental Results

The CPO framework was implemented using Qwen2.5-VL as the base model and evaluated against major benchmarks, including MIMIC-CXR and MS-CXR-T.

  • State-of-the-Art Classification: On the MS-CXR-T benchmark, the CPO-tuned model achieved 81.8% accuracy, representing a 12.6% improvement over the previous state-of-the-art (CoCa-CXR).
  • Enhanced Report Generation: The model demonstrated significant gains in narrative alignment, with a 16.5% improvement in BLEU-4 and a 34.8% boost in METEOR scores.
  • Zero-Shot Generalization: In tests across six external datasets (including Open-I and PadChest), CPO consistently outperformed competitors, suggesting that the model learned robust medical reasoning rather than simple pattern memorization.
  • Ablation Studies: The inclusion of CPO provided a 4.5% gain in accuracy, significantly outperforming standard Chain-of-Thought fine-tuning, which only yielded a 1.1% improvement.

Discussion and Future Work

The success of CPO suggests that for highly specialized domains, understanding what a condition is not is as vital as understanding what it is. By managing the causal path of reasoning, the researchers have demonstrated a path toward more reliable MLLMs in clinical settings.

However, limitations remain. The generation of high-quality counterfactuals is currently a computational bottleneck and relies heavily on the availability of a high-quality "LLM Expert" and structured knowledge graphs. Future research will focus on improving the efficiency of counterfactual generation and exploring the framework's applicability to other complex fields, such as legal reasoning or software engineering, where concept drift similarly threatens model reliability.

Summary updated

Abstract

This paper uncovers a critical yet overlooked phenomenon in multi-modal large language models (MLLMs), especially for chest diagnosis: detrimental concept drift within chain-of-thought (CoT) reasoning during non-stationary reinforcement fine-tuning (RFT), where reasoning token distributions evolve unpredictably, thereby introducing significant biases in final predictions. To address this, we are pioneers in establishing the theoretical bridge between concept drift theory and RFT processes by formalizing CoT's autoregressive token streams as non-stationary distributions undergoing arbitrary temporal shifts. Leveraging this framework, we propose a novel autonomous counterfact-aware RFT that systematically decouples beneficial distribution adaptation from harmful concept drift through concept graph-empowered LLM experts generating counterfactual reasoning trajectories. Our solution, Counterfactual Preference Optimization (CPO), enables autonomous and stable RFT in non-stationary environments, particularly within the medical domain, through custom-tuning of counterfactual-aware preference alignment. Extensive experiments demonstrate our superior performance of robustness, generalization and coordination within RFT. Besides, we also contribute a large-scale dataset CXR-CounterFact (CCF), comprising 320,416 meticulously curated counterfactual reasoning trajectories derived from MIMIC-CXR. Our code and data are public at: https://github.com/XiaoyuYoung/CPO.
BibTeX
@inproceedings{yang2025walking,
  author = {Xiaoyu Yang and Jie Lu and En Yu},
  title = {{Walking the Tightrope: Autonomous Disentangling Beneficial and Detrimental Drifts in Non-Stationary Custom-Tuning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=1BAiQmAFsx},
}
Figure