Conflict-Aware Outer Steps for Mixed RL and Supervised Multi-Task Training

When a single model is trained on several tasks at once, the tasks often pull its shared weights in opposite directions. The standard fix is to compare the tasks' gradients at every update and remove the parts that conflict. That works on one machine. It breaks down when training is spread across machines, because each comparison forces the machines to exchange full gradients at every step. It breaks down further when one of the tasks is trained by reinforcement learning, meaning trial-and-error sampling of outputs scored by a reward. Sampling is slow and sequential, so the supervised tasks sit idle at every lockstep update while the sampling task catches up. We propose to run each task on its own group of workers, at that task's natural speed and without communication, and to resolve gradient conflicts only at infrequent synchronization rounds. At each round, the net weight change produced by each task is projected away from the others before the changes are combined.

Our test case is noisy social media text. One task detects offensive or aggressive posts, which is cheap supervised classification. The other task rewrites a post into standard spelling ("text normalization"). We fine-tune that rewriter with a policy-gradient method rewarded by edit distance to the reference, which is expensive because it must generate text. The claim is that conflict removal applied once per round, rather than once per step, recovers most of the accuracy benefit of per-step conflict removal. It should do so at the communication cost of low-communication distributed training and without forcing the fast task to wait for the slow one. If that holds, multi-task RL becomes practical in exactly the setting where its cost is highest.

Core Claim

Three bodies of work meet here without touching. The first is gradient surgery. Gradient Surgery for Multi-Task Learning (PCGrad) detects conflicting task gradients ($g_i \cdot g_j < 0$) and projects one onto the normal plane of the other. It reports large gains, for example raising Meta-World MT50 success from under 40% to about 70%, but it assumes a centralized loop with separate backward passes per task at every step. RotoGrad shares that per-step assumption.

The second is low-communication training. DiLoCo, studied in Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo, lets replicas take $H$ local steps and then averages their weight changes as a "pseudo-gradient" for an outer optimizer. It cuts bandwidth 10–100x while keeping compute utilization above 80%, but it has only been tested on single-objective pretraining on clean text.

The third is MERIT, from Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging. It is the closest existing work. MERIT estimates task conflicts once, on 200 calibration samples, splits conflicting data onto independent branches, and merges them once at the end. On Qwen2.5-VL-3B this raised an 8-benchmark average from 54.3 to 57.0. However, MERIT is supervised-only, and its conflict handling is a one-time partition rather than a repeated correction.

Our method differs from all three. Conflicts are measured and removed on the per-task pseudo-gradients at every outer round. Rounds are defined by wall-clock time rather than step count, so a sampling-bound task and a teacher-forced task each make as much progress as they can between rounds. Let $\theta$ be the shared weights at the start of a round. Let $\Delta_t = \theta - \theta_t$ be the change, as a pseudo-gradient, produced by the worker group for task $t$, which ends the round at local weights $\theta_t$. The outer update first applies

$$\tilde\Delta_i = \Delta_i - \frac{\min(0,\ \Delta_i \cdot \Delta_j)}{\|\Delta_j\|^2}\,\Delta_j ,$$

which removes only the conflicting component. It then feeds $\sum_t \tilde\Delta_t$ to a Nesterov-momentum outer optimizer, as in DiLoCo. Communication per round equals one DiLoCo round per task group. The projection costs one dot product per task pair on vectors that are already being exchanged.

The Application Regime and Why It Stresses the Method

Social media text supplies a pair of tasks with a known mutual benefit and a plausible conflict. Aggressive Language Detection with Joint Text Normalization via Adversarial Multi-task Learning motivates the pairing. It showed that joint training of detection and normalization beats a normalize-then-classify pipeline (64.0 F1 on TRAC Facebook, 53.6 on TRAC Twitter). It needed a shared/private split with adversarial and orthogonality losses to keep the tasks from interfering, which is direct evidence of conflict. Its normalizer was trained only with token-level teacher forcing.

Historical Text Normalization with Delayed Rewards supplies the RL ingredient. It fine-tunes a likelihood-pretrained normalizer with REINFORCE using negative Levenshtein distance as a delayed reward. This cut errors by 31% on Slovene and 20% on English historical text, with no learned critic. That design matters for overhead: it needs no separate value network or reward model. It therefore avoids most of the four-model memory burden that P-TA and Getting More Juice Out of the SFT Data describe for PPO-style pipelines. What remains is the generation bottleneck that Is Reinforcement Learning (Not) for Natural Language Processing identifies, and that bottleneck is the one our scheduling targets.

Enhancing BERT for Lexical Normalization constrains the design in two ways. It shows that over 90% of tokens need no change, which raises a risk that a sequence-level reward mostly teaches copying. It also shows that noise-focused upsampling is needed to fight that tendency, so we keep it in the likelihood warm-up.

Research Program

We build one shared encoder-decoder (T5-base scale) with two outputs. The first is a classification head on the encoder for offensive and aggressive language. It is trained on OLID/SOLID from OffensEval-2020, TRAC, and the HSOL/OLI/KTC suite. The second is the decoder for normalization. It is warmed up with likelihood training on WNUT 2015 (lexnorm15) with noise-focused upsampling, then fine-tuned with REINFORCE using negative Levenshtein reward against gold targets.

Detection workers and normalization workers form separate groups that start each round from the same $\theta$. Rounds last a fixed wall-clock interval $T$, so the normalization group completes far fewer steps than the detection group. Because the groups take different numbers of steps, their pseudo-gradients differ in magnitude. We study the magnitude imbalance that PCGrad calls one leg of its "tragic triad" by comparing raw pseudo-gradients against pseudo-gradients rescaled by local step count.

Training runs on 16 GPUs split into two simulated "islands" with throttled inter-island bandwidth. This makes communication cost a measured quantity rather than an assumption. We log four quantities. The first is the cosine between $\Delta_{\text{det}}$ and $\Delta_{\text{norm}}$ at every round, which tells us whether conflicts actually occur at this granularity. The second is idle time per worker. The third is bytes transmitted. The fourth is task metrics as a function of wall-clock time and of FLOPs.

Baselines

The naive reference point trains each task in isolation: a detector alone and a likelihood-only normalizer alone, each with the same backbone, data, and per-task compute budget. Every multi-task variant must beat this to show positive transfer. We also run a pipeline variant, in which the normalizer feeds the detector, because Aggressive Language Detection with Joint Text Normalization via Adversarial Multi-task Learning reports that it suffers from error propagation and we want to confirm this with our backbone.

The strongest published systems give external anchors. For normalization the anchor is Enhancing BERT for Lexical Normalization, at 79.28 F1 on lexnorm15. For OffensEval-2020 English Sub-task A, Kungfupanda reports 91.51 macro-F1 and 0.8382 F1 on OLID for its BERT multi-task ensemble. For TRAC, the adversarial joint model reports 64.0 F1 (Facebook) and 53.6 F1 (Twitter), plus an average of 90.5 F1 over OLI, KTC, and HSOL. That paper's standalone lexnorm15 F1 is not isolated in our file. We treat these as reference points rather than direct competitors, because backbones differ.

The direct competitors share our backbone, data, total GPU-hours, and tuning budget, with the same number of learning-rate and $H$/$T$ trials for each. The first is synchronous multi-task training with per-step all-reduce and the same RL objective. It is the most accurate per step, but it pays full communication and idle cost. The second adds per-step PCGrad to that synchronous setup, which is the accuracy ceiling for conflict removal. The third is plain DiLoCo with task-specialized groups and simple averaging, which forms the ablation that isolates the projection. The fourth is MERIT-style one-shot splitting and merging, run with its 200-sample calibration and a token-weighted final merge.

Our method beats the field if two things hold at equal wall-clock and equal bytes. It must exceed plain DiLoCo averaging and MERIT on both tasks. It must also land within roughly one F1 point of per-step synchronous PCGrad on both tasks, while using at least an order of magnitude less inter-island communication and substantially less idle time.

Why not just…?

Why not just train both tasks jointly with supervised learning on one cluster, as the adversarial joint model does? This is the default, and it may be enough on accuracy. Its weakness is not quality but the fact that it cannot absorb an RL task without lockstep stalls, and it needs extra adversarial machinery to manage interference. The synchronous baseline with and without the RL objective measures both the accuracy gain from RL and the idle-time cost of adding it synchronously. If RL adds nothing on lexnorm15, the motivation for heterogeneous scheduling weakens, and we will report that result plainly.

Why not just skip reinforcement learning, since likelihood training is parallel and cheap? This is a real threat. Social media text is mostly copyable, and the gains reported by Historical Text Normalization with Delayed Rewards came from historical spelling with long, unseen words and an LSTM model. We test the question directly by comparing likelihood-only and likelihood-plus-REINFORCE normalizers in isolation, broken down by word length and by out-of-vocabulary status. If REINFORCE gives no gain, the study still answers the distributed-conflict question with two supervised tasks, but the "RL makes lockstep wasteful" argument would then rest only on the idle-time measurements.

Why not just use MERIT, which already splits conflicting tasks and merges once? MERIT is the strongest prior method, and with only two tasks starting from a pretrained basin, one-shot merging might suffice. Our bet is that conflict between detection and RL normalization changes during training, especially as the policy drifts from its likelihood initialization. A single early estimate from calibration samples would then miss it, while per-round projection would track it. The round-by-round cosine log tests this directly. If cosines stay non-negative or stable, MERIT should match us, and in that case the remaining contribution is only the time-based scheduling.

Why not just compress gradients and keep per-step PCGrad? Compression would preserve per-step conflict removal. However, Saurabh Agarwal et al. show that compression consumes GPU cycles and often forces less scalable collective operations. Compression also does nothing about the fast task waiting for rollouts. We include 4-bit-compressed synchronous PCGrad as a secondary comparison on wall-clock time.

Decisive Checks

The first check is whether conflict exists at round granularity. If the cosine between pseudo-gradients is rarely negative, the projection is inert and the idea reduces to DiLoCo. The second check is a sweep over round length $T$. Accuracy relative to per-step PCGrad should degrade smoothly as $T$ grows, and the useful region should begin at $T$ values large enough to cut communication 10x or more. The third check is the ablation of projection against plain averaging at matched $T$ on lexnorm15 F1 and OffensEval Sub-task A macro-F1. The fourth is an over-correction audit of normalizer outputs on slang, brand names, and obfuscated slurs, because both conflict removal and reward optimization could push the rewriter to rewrite valid tokens. The fifth is a measurement of idle time and bytes per round, which is what makes the systems claim falsifiable rather than asserted.