CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning

Jixuan Leng, Si Si, Fei Wang, Cho-Jui Hsieh, Jiong Zhang, Hsiang-Fu Yu, Inderjit S. DhillonarXiv 2026 · arXiv

TL;DR

CONCUR is a novel advantage estimator for multi-reward Reinforcement Learning (RL) that prevents auxiliary rewards—like length penalties—from overriding a task's primary goal. By using a projection based on isotonic regression, it ensures that auxiliary rewards respect the primary reward's ordering, leading to significant performance gains across math, coding, and writing tasks.

What Problem Does This Paper Address?

When training Large Language Models (LLMs) using RL, researchers often combine multiple reward signals. A primary reward measures the core objective (e.g., "Is the math answer correct?"), while auxiliary rewards encourage secondary behaviors (e.g., "Keep the response concise").

To prevent the model from "cheating," these rewards are often gated: an auxiliary reward shouldn't be high if the primary reward is low. However, modern RL algorithms like GRPO or GDPO use "group-relative" advantage estimation, which standardizes each reward component independently within a group of responses. The authors identify two critical failure modes caused by this standardization:

  1. Against the Order: An auxiliary reward can be so strong that it reverses the primary reward's preference—for example, favoring a short, wrong answer over a slightly longer, correct one.
  2. At the Ceiling: When all responses in a group are correct (reaching the "ceiling" of the primary reward), the primary reward's standardized score becomes zero. The auxiliary reward then becomes the only signal, forcing the model to optimize for length or style rather than maintaining the quality it just achieved.

Methods

The authors propose CONCUR (CONCordant Underlying Rewards), an advantage estimator that replaces standard auxiliary scores with a projection onto the closest scores that "concur" with the primary reward.

Figure 1: Left: GDPO standardizes reward components in a group and sums them at the declared weights, and the insets show where an auxiliary departs from its primary. Right: CONCUR projects each gated auxiliary’s scores onto the closest ones that concur with the primary, and in CONCUR-A earned trust decides whether a ceiling is tied.

CONCUR enforces two main constraints:

  • Order Constraint: A response with a higher primary reward cannot receive a lower auxiliary score than one with a lower primary reward.
  • Ceiling Constraint: Responses that all reach the primary reward's maximum (the ceiling) must receive equal auxiliary scores.

The paper introduces two variants:

  • CONCUR-T (Tied): Always enforces both constraints. This is the recommended default.
  • CONCUR-A (Adaptive): Uses a rule called "Earned Trust." If an auxiliary reward consistently agrees with the primary reward's ordering on non-ceiling responses, the algorithm "trusts" it enough to let it separate responses even at the ceiling.

Technically, CONCUR implements these constraints efficiently using the Pool-Adjacent-Violators Algorithm (PAVA), a standard method for isotonic regression.

Datasets

The models were trained on a four-task mixture:

  • Math: Correctness (binary) gating a length reward.
  • Code: Unit-test pass rate gating a length reward.
  • Instruction Following (IF): Rule-checked criteria gating a length reward.
  • Writing: Judged criteria gating both a length reward and a Reward Model (RM) score.

Evaluation spanned 15 benchmarks, including AIME24/25, MATH-500, HumanEval+, MBPP+, IFEval, and WritingBench.

Key Insights and Ideas

  • Standardization is a Double-Edged Sword: While group-relative normalization helps stabilize training, it can amplify tiny, irrelevant differences in auxiliary rewards (like a few characters of length) into massive advantage signals when the primary reward is tied.
  • The "Ceiling" is a Danger Zone: In tasks like Math, where rewards are often binary (correct/incorrect), models frequently reach the ceiling. Without CONCUR, the model spends its RL budget optimizing for brevity rather than mathematical robustness.
  • Smart Weighting, Not Constant Weighting: CONCUR effectively acts as a dynamic weighting scheme. It doesn't just turn off auxiliaries; it reshapes them group-by-group to ensure they never contradict the primary mission.
  • Earned Trust: The idea that an auxiliary reward must "prove" its alignment with the primary goal before being allowed to make fine-grained distinctions at the performance ceiling is a clever way to balance multiple objectives.

Key Results

CONCUR consistently outperformed standard estimators (GDPO, GRPO) and a "no-auxiliary" baseline across three different model families (Qwen3-4B, Qwen3-8B, and OLMo-3-7B).

Figure 3: Panels (a) to (c): each task group’s score and Avg minus those of GDPO. Panel (d): each gated auxiliary’s mean score over the passing responses, as the mean over the last 20 training steps minus the base policy’s level.
  • Average Gains: Both CONCUR variants scored more than 4 points higher on average than GDPO across all 15 benchmarks.
  • Math Performance: The gains were most dramatic in math, with improvements of 6.3 to 8.9 points over the baseline.
  • Writing Quality: Unlike the "no-auxiliary" baseline (which improved math but tanked writing scores), CONCUR improved math while also beating the baseline on writing tasks.
  • Training Stability: CONCUR maintained a higher cosine similarity between the total advantage and the primary reward throughout training, indicating a more focused optimization path.

Limitations and Future Work

  • Fixed Gating: The method assumes a pre-defined gating structure (e.g., length reward is gated by correctness). It doesn't learn the hierarchy of rewards automatically.
  • Single Mixture: The evidence is based on one specific four-task mixture. While diverse, further testing on different task combinations would strengthen the results.
  • Judge Dependency: For writing tasks, the gains were smaller when evaluated by a "second judge" model different from the one used in training, suggesting some degree of over-optimization to the specific training judge's preferences.
  • Future Directions: The authors suggest exploring how CONCUR could be applied to process-based rewards (PRMs) or more complex multi-objective RL scenarios where the priority between rewards might change over time.

Summary updated

Abstract

When a Large Language Model (LLM) is trained with Reinforcement Learning (RL) on multiple tasks, a task's primary reward can gate auxiliary rewards such as a length reward: on a failing response, an auxiliary cannot exceed the primary. However, group-relative advantage estimators treat a gated auxiliary like any other component, so it can reverse the primary's order, and at the primary's ceiling only auxiliaries order responses. Keeping gated auxiliaries hurts some tasks, while dropping them hurts others. We propose CONCUR, an advantage estimator that replaces the standardized scores of each gated auxiliary with the closest scores that satisfy two conditions: responses with higher primary rewards cannot receive lower scores, and responses at the primary's ceiling receive equal scores. CONCUR-T applies both conditions throughout training, and CONCUR-A lifts the second for an auxiliary that agrees with the primary's order more often than it reverses it among passing responses. On three models from two families and across math, code, instruction following, and writing, both variants score higher on average than all baselines and than when all the gated auxiliaries are dropped.
BibTeX
@misc{leng2026concur,
  author = {Jixuan Leng and Si Si and Fei Wang and Cho-Jui Hsieh and Jiong Zhang and Hsiang-Fu Yu and Inderjit S. Dhillon},
  title = {{CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning}},
  year = {2026},
  eprint = {2610.36437},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36437},
}
Figure