Transferability for General Reasoning: An Automated Curriculum for Multi-Domain LLM RL

RLxF @ ICML 2026 · OpenReview

In the race to build "General Purpose" reasoning models, researchers have moved beyond training on just math or just code. We now use multi-domain suites spanning science, logic, and table understanding. But here’s the catch: not all training data is created equal. Training an LLM on a specific math problem might help it solve physics, but training it on a niche logic puzzle might not help it with anything else.

A new paper, "Transferability for General Reasoning: An Automated Curriculum for Multi-Domain LLM RL," introduces a clever way to navigate this. Instead of sampling training data randomly or just focusing on what the model is currently "learning" best, the authors propose a curriculum that prioritizes tasks that provide the most "transferable" knowledge to the rest of the model’s repertoire.

TL;DR

The authors introduce Transfer-Aware Curriculum (TAC), a bandit-style system for Reinforcement Learning (RL) that automatically decides which domain to train on next. By analyzing gradient alignments, TAC prioritizes tasks that don't just improve the model locally, but provide "spillover" benefits to other reasoning domains, leading to significant accuracy gains with almost zero extra compute cost.

What Problem Does This Paper Address?

Reinforcement Learning with Verifiable Rewards (RLVR)—the technique behind models like DeepSeek-R1—is powerful but sensitive to the "curriculum" (the order and frequency of training tasks).

Currently, most researchers use two methods:

  1. Fixed/Random Sampling: Give the model a bit of everything.
  2. Learnability-based Curricula: Focus on tasks where the model’s performance is currently "in the zone"—not too easy, not too hard.

The problem? Learnability is blind to transferability. A model might spend all its time mastering a highly "learnable" but isolated task that offers zero benefit to its overall reasoning capabilities. This paper argues that we should prioritize "hubs"—tasks that, when learned, move the model's parameters in a direction that helps it solve many other types of problems.

Methods

The authors treat domain selection as a Multi-Armed Bandit problem. Each "arm" is a reasoning domain (e.g., Math, Code, Logic). To decide which arm to pull, TAC uses a composite signal:

  • Local Learnability ($L$): Derived from the "advantages" in the GRPO (Group Relative Policy Optimization) algorithm. If the model sees a wide range of rewards (some successes, some failures), the task is considered learnable.
  • Global Transferability ($T$): This is the novel contribution. The authors use random projections to compress the model's high-dimensional gradients into a small vector. They then calculate the cosine similarity between the gradient of the current task and the moving averages of gradients from all other tasks. If the directions align, the task is "transferable."

The final selection score is a mix: $S = \beta L + (1-\beta) T$. Interestingly, the authors found that a low $\beta$ (0.2) worked best, meaning transferability is actually a stronger signal for success than learnability.

Datasets

The study uses the GURU multi-domain reasoning suite, which includes:

  • Math: Competition-level problems (AIME, AMC).
  • Codegen: Programming tasks (LeetCode, MBPP).
  • Logic: Puzzles like ARC-AGI and Zebra puzzles.
  • Simulation: Predicting program inputs/outputs (Code I/O).
  • Table: Reasoning over structured data (HiTab, MultiHierTT).
  • STEM: Scientific reasoning (OpenScienceReasoning-2).

Evaluation was performed across 14 held-out benchmarks to ensure the model wasn't just memorizing the training set.

Key Insights and Ideas

  • The Math Paradox: One of the most surprising findings is that Math is one of the least transferable domains. While RL on math is popular, its gradients often point in "idiosyncratic" directions that don't help (and sometimes oppose) other domains.
  • Table Reasoning as a Hub: Conversely, training on Table Reasoning acts as a massive "hub," providing updates that align well with simulation, logic, and STEM tasks.
  • Efficiency is Key: Comparing full gradients for LLMs is computationally impossible. By using TRAK-style random projections, the authors calculated transferability with <1% wall-clock overhead. It’s essentially a "free" performance boost.
  • Dynamic Adaptation: Transferability isn't static. As the model gets better at coding, the "transfer value" of coding tasks might drop. TAC's bandit approach allows the curriculum to evolve in real-time as the model's "knowledge map" changes.

Key Results

The authors tested TAC on Qwen3-1.7B and Llama3.2-3B:

  • Superior Performance: TAC achieved the highest macro-averaged accuracy across both models, outperforming random sampling and learnability-only baselines (like SEC).
  • Significant Gains: On Qwen3-1.7B, TAC improved macro-averaged accuracy by 1.8 percentage points over the strongest baseline. On specific tasks like "Zebra puzzles," the gain was a massive +9.8 points over random sampling.
  • Robustness: Even when the training data was intentionally imbalanced (giving the model way too much math data), TAC was able to "correct" the course by down-weighting math once it realized the updates weren't transferring.

Limitations and Future Work

  • Verifiable Rewards Only: The current system relies on "verifiable rewards" (where a computer can check if the answer is 100% right). The authors acknowledge that extending this to "soft" rewards (like LLM-as-a-judge for creative writing) is a major next step.
  • Parameter Subsets: To save compute, they only analyzed gradients from the last 4 layers of the transformer. While this worked well, exploring whether earlier layers provide different transfer signals remains an open question.
  • Scaling: While they tested up to 4B parameters, seeing how these transfer "hubs" change in 70B+ parameter models would be highly valuable for the industry.

Summary updated

Abstract

Reinforcement learning with verifiable rewards (RLVR) has recently expanded from single-domain training to multi-domain reasoning suites spanning mathematics, programming, science, and related tasks. However, the training curriculum, i.e., how often each domain is sampled, is usually fixed or hand-tuned, despite evidence that reasoning skills transfer unevenly across domains. Existing learnability-based curricula adapt to where the policy is currently improving, but do not account for whether a gradient step on the selected domain also benefits the remaining domains. We propose Transfer-Aware Curriculum (TAC), a bandit-style online curriculum that prioritizes domains whose updates broadly transfer across the training suite. TAC reuses signals already produced during RL training: per-domain advantages estimate local learnability, while projected gradients from the GRPO step estimate cross-domain transferability through gradient-geometry alignment, adding negligible overhead (<1% wall-clock). Across a six-domain reasoning suite, TAC achieves the best macro-averaged accuracy on both Qwen3-1.7B and Llama3.2-3B, outperforming proportional random sampling, a hand-designed schedule, and a learnability-only bandit, with gains over the latter of up to 2.8 points (10% relative). Ablations show that removing transferability sharply degrades performance, and TAC remains robust under imbalanced training mixtures where learnability-only curricula over-commit to dominant domains. These results establish cross-domain transferability as an important signal for curriculum design in multi-domain RLVR.
BibTeX
@inproceedings{2026transferability,
  title = {{Transferability for General Reasoning: An Automated Curriculum for Multi-Domain LLM RL}},
  booktitle = {RLxF @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=8Y9pNLYiup},
}
Figure