TL;DR
CORD (Conflict-Aware Optimization for Joint RL and Distillation) is a general framework that prevents reinforcement learning (RL) and distillation gradients from canceling each other out during agent training. By selectively projecting conflicting gradients, CORD consistently improves performance across complex benchmarks like ALFWorld and WebShop, offering a plug-and-play boost for existing joint training recipes.
What Problem Does This Paper Address?
Training language-model agents is hard because environment rewards (RL) are often sparse—you only know if you succeeded at the very end. To help agents learn faster, researchers often use "self-distillation," where a "teacher" (often a version of the model with extra context) provides dense, token-level guidance.
The problem is that these two signals—the RL reward and the teacher's guidance—don't always agree. Their gradients can point in opposite directions, a phenomenon called gradient conflict. When you simply add these conflicting gradients together (the standard "DirectSum" approach), they partially cancel out, leading to stalled learning and lost optimization progress. Simply tuning a mixing coefficient ($\lambda$) doesn't solve this; it just scales the conflict without removing the opposing components.
Methods
The authors introduce CORD, which treats joint training as a multi-objective optimization problem with a "protected" objective.
- Identify Conflict: CORD calculates the cosine similarity between the RL gradient ($u$) and the distillation gradient ($v$). A negative cosine indicates conflict.
- Selective Projection: When conflict is detected, CORD keeps the "protected" gradient intact and modifies the other (the "candidate"). It projects the candidate gradient onto the orthogonal complement of the protected one. This effectively "removes" the component that was pushing against the protected objective.
- Flexible Gating: CORD offers two modes:
- Conflict-only: Only projects when gradients point in opposite directions.
- Always-project: Projects even when gradients are aligned, ensuring they remain orthogonal.
- Mini-batch Aggregation: Unlike some prior methods that project at every step, CORD accumulates gradients over a full mini-batch before projecting once. This makes it computationally efficient and more stable.
Datasets
The researchers evaluated CORD using the SDAR agent suite with the Qwen2.5-3B-Instruct model on two primary benchmarks:
- ALFWorld: A synthetic environment where agents must complete household tasks (e.g., "put a hot egg in the fridge") through multi-step text interactions.
- WebShop: A simulated e-commerce site where agents must follow complex instructions to find and buy specific products based on attributes like color, size, and price.
Key Insights and Ideas
- Conflict is Frequent: In WebShop training logs, nearly 48.2% of mini-batch gradients showed conflict. This isn't a rare edge case; it's a fundamental hurdle in joint training.
- Protection Matters: You have to choose which objective to prioritize. Protecting RL (ensuring rewards are never compromised) vs. protecting distillation (ensuring the teacher is always followed) leads to different agent behaviors.
- Behavioral Shifts: CORD doesn't just change numbers; it changes how agents "act." For example, protecting RL in WebShop tends to make agents more efficient (fewer steps), while protecting distillation can improve attribute completion but sometimes leads to more repetitive actions.
- Synergy with Existing Methods: CORD isn't a new training "recipe" but a better way to combine ingredients. It was successfully added to three different existing methods (Skill-SD, GRPO+OPSD, and SDAR), improving all of them.
Key Results
CORD provided significant "incremental effectiveness" over base methods:
- ALFWorld: Success rates jumped by 1.5 to 15.9 percentage points. The GRPO+OPSD method saw the biggest boost, rising from 65.0% to 80.9% success.
- WebShop: CORD improved success rates for two out of three host recipes. For the SDAR method, success rose from 60.7% to 64.6%.
- Efficiency: Qualitative analysis showed that CORD-trained agents were better at fulfilling all requested product attributes while avoiding redundant steps compared to unprojected baselines.
Limitations and Future Work
- Sensitivity to Configuration: The authors found that the choice of which objective to protect and which "gate" to use is highly consequential. There is no "one-size-fits-all" setting; it depends on the specific task and the nature of the distillation loss.
- Computational Overhead: While CORD is $O(d)$ (linear with respect to parameters), it does require storing an additional gradient buffer, which increases memory usage.
- Future Directions: The paper suggests that future work could explore dynamic ways to switch the "protected" objective during training or extending the framework to handle more than two conflicting objectives simultaneously.
Summary updated