CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents

Wenhao Yu, Zhengcheng Wang, Jiahong Liu, Zexuan Qiu, Minda Hu, Shaohang Wei, Jiafei Lyu, Junyan Lyu, Haobo Fu, Irwin KingarXiv 2026 · arXiv

TL;DR

CORD (Conflict-Aware Optimization for Joint RL and Distillation) is a general framework that prevents reinforcement learning (RL) and distillation gradients from canceling each other out during agent training. By selectively projecting conflicting gradients, CORD consistently improves performance across complex benchmarks like ALFWorld and WebShop, offering a plug-and-play boost for existing joint training recipes.

What Problem Does This Paper Address?

Training language-model agents is hard because environment rewards (RL) are often sparse—you only know if you succeeded at the very end. To help agents learn faster, researchers often use "self-distillation," where a "teacher" (often a version of the model with extra context) provides dense, token-level guidance.

The problem is that these two signals—the RL reward and the teacher's guidance—don't always agree. Their gradients can point in opposite directions, a phenomenon called gradient conflict. When you simply add these conflicting gradients together (the standard "DirectSum" approach), they partially cancel out, leading to stalled learning and lost optimization progress. Simply tuning a mixing coefficient ($\lambda$) doesn't solve this; it just scales the conflict without removing the opposing components.

Figure 1. From gradient conflict to improved agent training. Left, observations (a) and (c): cosine quantiles of conflicting WebShop/Qwen3-1.7B gradient records (Qwen Team, 2025), before and after analytic CORD projection. Middle, observation (b): first-order RL-surrogate progress relative to RL alone on the same log, evaluated at 𝜆= 0.005 with RL protection. Right: preliminary WebShop/3B training success with 𝑘3, 𝜆= 0.001, and conflict-only distillation protection. The right panel shows iterations 75–148 after 21-iteration smoothing. Curves average two training seeds and bands show their range. Appendix C.1 gives measurement and smoothing details.

Methods

The authors introduce CORD, which treats joint training as a multi-objective optimization problem with a "protected" objective.

  1. Identify Conflict: CORD calculates the cosine similarity between the RL gradient ($u$) and the distillation gradient ($v$). A negative cosine indicates conflict.
  2. Selective Projection: When conflict is detected, CORD keeps the "protected" gradient intact and modifies the other (the "candidate"). It projects the candidate gradient onto the orthogonal complement of the protected one. This effectively "removes" the component that was pushing against the protected objective.
  3. Flexible Gating: CORD offers two modes:
  4. Conflict-only: Only projects when gradients point in opposite directions.
  5. Always-project: Projects even when gradients are aligned, ensuring they remain orthogonal.
  6. Mini-batch Aggregation: Unlike some prior methods that project at every step, CORD accumulates gradients over a full mini-batch before projecting once. This makes it computationally efficient and more stable.
Figure 2. Geometric intuition: CORD keeps the protected gradient $u$ intact and projects the conflicting gradient $v$ to $\tilde{v}$, ensuring the final update $u + \tilde{v}$ doesn't hinder progress on the protected task.

Datasets

The researchers evaluated CORD using the SDAR agent suite with the Qwen2.5-3B-Instruct model on two primary benchmarks:

  • ALFWorld: A synthetic environment where agents must complete household tasks (e.g., "put a hot egg in the fridge") through multi-step text interactions.
  • WebShop: A simulated e-commerce site where agents must follow complex instructions to find and buy specific products based on attributes like color, size, and price.

Key Insights and Ideas

  • Conflict is Frequent: In WebShop training logs, nearly 48.2% of mini-batch gradients showed conflict. This isn't a rare edge case; it's a fundamental hurdle in joint training.
  • Protection Matters: You have to choose which objective to prioritize. Protecting RL (ensuring rewards are never compromised) vs. protecting distillation (ensuring the teacher is always followed) leads to different agent behaviors.
  • Behavioral Shifts: CORD doesn't just change numbers; it changes how agents "act." For example, protecting RL in WebShop tends to make agents more efficient (fewer steps), while protecting distillation can improve attribute completion but sometimes leads to more repetitive actions.
  • Synergy with Existing Methods: CORD isn't a new training "recipe" but a better way to combine ingredients. It was successfully added to three different existing methods (Skill-SD, GRPO+OPSD, and SDAR), improving all of them.

Key Results

CORD provided significant "incremental effectiveness" over base methods:

  • ALFWorld: Success rates jumped by 1.5 to 15.9 percentage points. The GRPO+OPSD method saw the biggest boost, rising from 65.0% to 80.9% success.
  • WebShop: CORD improved success rates for two out of three host recipes. For the SDAR method, success rose from 60.7% to 64.6%.
  • Efficiency: Qualitative analysis showed that CORD-trained agents were better at fulfilling all requested product attributes while avoiding redundant steps compared to unprojected baselines.
Figure 4. Qualitative comparison in WebShop: The unprojected policy fails to select the color, while CORD-protected policies succeed. Note how protecting RL leads to a more efficient 6-step path compared to the 10-step path when protecting distillation.

Limitations and Future Work

  • Sensitivity to Configuration: The authors found that the choice of which objective to protect and which "gate" to use is highly consequential. There is no "one-size-fits-all" setting; it depends on the specific task and the nature of the distillation loss.
  • Computational Overhead: While CORD is $O(d)$ (linear with respect to parameters), it does require storing an additional gradient buffer, which increases memory usage.
  • Future Directions: The paper suggests that future work could explore dynamic ways to switch the "protected" objective during training or extending the framework to handle more than two conflicting objectives simultaneously.

Summary updated

Abstract

Joint reinforcement learning and self-distillation combines task rewards with dense teacher guidance for agent training, but their gradients can partly cancel. A mixing coefficient controls relative scale without selectively removing opposing components. We present CORD, a general framework for conflict-aware optimization of two objectives. CORD separates the choice of protected objective from the condition that triggers projection, and combines the gradients once per optimizer mini-batch. A nearest-point formulation yields a closed-form update and characterizes its first-order effect on the protected objective before optimizer transformations. Gradient diagnostics connect conflict in agent training to lost local optimization progress and motivate selective correction. As a gradient-combination mechanism, CORD can be added to different joint training recipes while retaining their objectives. Across three such recipes, it improves ALFWorld success by 1.5-15.9 percentage points. On WebShop, it improves two of the three pairs, with SDAR rising from 60.7% to 64.6% success. Our analyses explain why protection direction and projection gate must be chosen together, and why the distillation objective and its mixing weight remain consequential after projection.
BibTeX
@misc{yu2026cord,
  author = {Wenhao Yu and Zhengcheng Wang and Jiahong Liu and Zexuan Qiu and Minda Hu and Shaohang Wei and Jiafei Lyu and Junyan Lyu and Haobo Fu and Irwin King},
  title = {{CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents}},
  year = {2026},
  eprint = {2610.29965},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.29965},
}
Figure