When training machine learning models to reason, a natural intuition suggests matching the complexity of the training data to the target task: train on simple problems for simple tasks, and hard problems for hard tasks. However, a growing body of research reveals a striking asymmetry in how models transfer knowledge across difficulty levels.
While training models on simple tasks rarely equips them to handle complex, multi-step scenarios, training them on highly complex tasks regularly enhances their performance on simpler versions of those same problems. This phenomenon—often called downward generalization or positive backward generalization—suggests that exposure to high-complexity data forces models to internalize robust, foundational schemas that simpler data fails to activate.
The Asymmetry of Cross-Difficulty Transfer
The core of downward generalization lies in an evolutionary asymmetry between forward and backward task transfer. In Exploring Compositional Generalization of Large Language Models, Haoran Yang et al. formalize this dynamic by evaluating how models handle compositional instructions. They observe that training on multi-step instructions consistently improves performance on atomic sub-steps (positive backward generalization), whereas training exclusively on simple tasks fails to scale upward (negative forward generalization).
This asymmetry is not a quirk of prompt formatting. As Jing Zhou et al. demonstrate in Not All Tasks Are Born Equal: Understanding Zero-Shot Generalization, the generalization benefit of a training task is fundamentally tied to its inherent reasoning complexity rather than its linguistic structure. Simply reformatting low-complexity tasks into complex-looking multiple-choice questions does not trigger downward transfer; the underlying logic itself must be rigorous.
Forward Transfer (Fails): [Simple Tasks] --------X--------> [Complex Tasks]
Backward Transfer (Works): [Complex Tasks] ==================> [Simple Tasks]
When models are restricted to simple training data, they often rely on superficial heuristics or memorization. Chulin Xie et al. explore this in On Memorization of Large Language Models in Logical Reasoning, finding that while models heavily memorize logical puzzles during training, exposure to higher-order constraints acts as a catalyst. It forces the model to move past rote memorization and develop scalable reasoning patterns that naturally apply to lower-order variants.
Why Complexity Forces Robust Representations
Why does harder training make a model better at easier tasks? The answer lies in how structural constraints shape internal representations. Complex tasks act as a mathematical superset of simpler tasks, requiring the model to learn fundamental mechanics like state tracking, structural decomposition, and error correction.
Overcoming Supervision Shortcuts
In simple tasks, models can easily find "supervision shortcuts"—spurious correlations that yield the correct answer without requiring true logical execution. Arvid Frydenlund investigates this in Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More. By training models on complex branching tree structures, the author forces them to adopt generalized decomposition strategies. Consequently, the models successfully generalize downward to solve linear path-finding tasks that they paradoxically fail to learn when trained on the linear paths directly.
Building Foundational Schemas
This structural enforcement spans multiple domains:
- Logical Reasoning: In Pushing LLMs to Their Logical Reasoning Bound: The Role of Data Reasoning Intensity, Zhen Bi et al. show that prioritizing data with high logical density builds a foundational schema. This schema inherently stabilizes and improves accuracy on low-complexity tasks.
- Long-Horizon Planning: Changliu Liu et al. demonstrate in Training Task Reasoning LLM Agents for Multi-turn Task Planning via Single-turn Reinforcement Learning that models exposed to extended planning sequences develop the reasoning depth necessary for multi-step autonomy, which easily translates down to short-horizon tasks. Similar planning advantages are detailed by Wenjun Li et al. in Unlocking the Planning Capabilities of Large Language Models with Maximum Diversity Fine-tuning.
- Physical and Visual Intuition: This principle extends beyond text. In Visual Stability Prediction and Its Application to Manipulation, Wenbin Li et al. show that training neural networks to predict the stability of highly complex block towers encourages the learning of robust physical support features, drastically improving performance on simple configurations.
Stabilizing Performance and Universal Templates
Downward generalization also solves a common failure mode in large language models: the "hard-to-easy inconsistency," where a model capable of solving intricate problems paradoxically trips over simpler versions due to minor distractions.
Zhe Yang et al. study this phenomenon in Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?. The authors find that exposing models to rigorous reasoning requirements—either via high-complexity training data or difficult few-shot prompts—stabilizes their performance and mitigates their sensitivity to basic distractions.
This prompting benefit is thoroughly mapped out by Yao Fu et al. in Complexity-Based Prompting for Multi-step Reasoning. They demonstrate that few-shot prompts containing highly intricate reasoning steps provide a superior procedural template. The model internalizes this comprehensive procedure and applies it universally, executing simple tasks with much higher fidelity than if it had been prompted with simple examples.
[Complex Prompting Template]
└── Internalized Comprehensive Procedure
├── Applied to Complex Task -> High Success
└── Applied to Simple Task -> Stabilized Performance (No Distractions)
This stabilization effect manifests across diverse architectures and modalities. For instance, Hao Cheng et al. show in Open-domain Question Answering via Chain of Reasoning over Heterogeneous Knowledge that multi-hop reasoning frameworks serve as a robust superset for basic information retrieval, handling single-step queries without degradation. In multimodal settings, Tao Zhang et al. note in MMhops-R1: Multimodal Multi-hop Reasoning that training on complex multi-hop visual reasoning tasks directly enhances basic single-hop query performance.
Practical Implications for Model Optimization
Understanding that complexity drives robust generalization allows for more efficient model training and compression pipelines. Rather than gathering massive volumes of diverse, low-level data, developers can focus on high-density, structurally intricate datasets.
In SQL-PaLM: Improved large language model adaptation for Text-to-SQL, Ruoxi Sun et al. demonstrate that training on text-to-SQL datasets with massive schemas and intricate logic yields excellent generalization to simpler benchmarks, whereas the reverse training pipeline is highly ineffective. Similarly, when compressing models, Aoming Liu et al. show in Beyond Token Pruning: Operation Pruning in Vision-Language Models that pruning policies optimized on high-complexity, text-intensive tasks generalize significantly better to simpler tasks, because difficult benchmarks are inherently better at identifying which computational operations are truly essential.
A Counter-Perspective: The Latent Activation of Easy Data
While the dominant trend emphasizes training on hard data, it is worth noting a nuance identified by Mohit Bansal et al. in The Unreasonable Effectiveness of Easy Training Data for Hard Tasks. The authors show that under certain conditions, fine-tuning on easy data can recover a massive portion of the performance gained from hard training data by activating latent capabilities already present in pre-trained models.
However, when it comes to teaching models new structural logic, avoiding shortcuts, and stabilizing performance against basic distractions, the consensus across the literature remains clear: exposing models to the deep constraints of high-complexity scenarios is the most reliable way to build a robust foundation that generalizes downward.
Key Foundational Work
Visual Stability Prediction and Its Application to Manipulation (2016) — contributed 1 concept elements to this direction.
Open-domain Question Answering via Chain of Reasoning over Heterogeneous Knowledge (2022) — contributed 1 concept elements to this direction.