Abstract
Gradual stacking is a common technique used to accelerate the training of Large Language Models (LLMs) by progressively increasing model depth. While its efficiency benefits are well-documented, its impact on the qualitative capabilities of the resulting models has remained under-explored. This paper introduces MIDAS (MIDdle grAdual Stacking), a novel stacking strategy that replicates middle layers rather than final layers. We demonstrate that MIDAS not only provides up to a 40% training speedup but also induces a specific inductive bias that significantly enhances a model's reasoning and mathematical abilities. Crucially, these gains persist even when comparing models with identical pretraining perplexity, suggesting that stacking fundamentally alters the model's internal processing logic in favor of algorithmic reasoning.
Introduction
The computational cost of training state-of-the-art Large Language Models (LLMs) is a significant barrier to research and development. "Gradual stacking" has emerged as a popular mitigation strategy: a smaller model is trained first, and its layers are then duplicated to initialize a deeper, more powerful architecture. While this "growth spurt" approach is known to save compute, it raises a fundamental question: does the path taken to reach a certain level of pretraining loss affect the model's downstream intelligence?
Our research investigates the gap between pretraining perplexity (the ability to predict the next token) and downstream skill (the ability to solve complex problems). We find that not all training trajectories are equal. Specifically, we show that stacking creates an inductive bias that makes models "smarter" for the same amount of "knowledge," particularly in domains requiring multi-step logic.
MIDAS: Middle Gradual Stacking
Standard stacking techniques typically replicate the final layers of a Transformer to increase depth. However, we argue that this is suboptimal. In a Transformer architecture, the initial and final layers often develop specialized roles related to input processing and output generation, respectively. Replicating these specialized layers can disrupt the model's structural integrity.
We propose MIDAS (MIDdle grAdual Stacking), which focuses on duplicating the middle blocks of the network. Middle layers are functionally more similar to one another and act as the core "processing engine" of the model. By expanding this middle section, MIDAS preserves the specialized roles of the "sandwich" ends while increasing the model's internal computational capacity.
To manage the transition between depths, we utilize PROP-α schedules. These schedules determine the proportion of training tokens allocated to each depth stage, ensuring the model spends sufficient time at full depth to stabilize and refine its parameters.
Experimental Results
We evaluated MIDAS across models ranging from 1B to 8B parameters, trained on a 500B token mixture of C4, Wikipedia, Arxiv, and Github. Our evaluation suite covered closed-book QA, open-book QA, math word problems (GSM8k, SVAMP), and a new set of "Reasoning Primitives" designed to isolate skills like variable assignment and pattern tracking.
The Reasoning Inductive Bias
The most striking finding is that MIDAS models consistently outperform traditionally trained baselines on reasoning tasks, even when their validation perplexity is identical. This indicates that stacking induces a qualitative shift in how the model processes information, favoring algorithmic-like logic over simple pattern matching.
As shown in the results, the gains are most pronounced in "reasoning primitives" and "math word problems." For instance, an 8B MIDAS model achieved 15.2% accuracy on GSM8k compared to 12.3% for the baseline. When arithmetic errors were corrected using an external calculator, the gap widened, proving that the model's underlying logical framework had improved.
Efficiency and Performance
Beyond qualitative improvements, MIDAS is highly efficient. It achieves up to a 40% speedup in training time while matching or exceeding the quality of a model trained from scratch at full depth. In the 1B parameter range, MIDAS reached an average accuracy of 26.8% across 15 tasks, significantly higher than the 24.0% achieved by the standard baseline.
Discussion: The Looped Model Connection
Why does stacking improve reasoning? We hypothesize that stacking increases the similarity between adjacent layers. This effect causes the Transformer to behave more like a "looped model," where weights are shared across layers. Theoretically, looped models are better suited for simulating iterative algorithms, which are essential for multi-step reasoning and compositional tasks. Our analysis of layer similarity heatmaps supports this conjecture, showing higher inter-layer correlation in MIDAS-trained models.
Conclusion and Future Work
MIDAS demonstrates that gradual stacking is more than just a training optimization; it is a method for shaping a model's inductive bias. By focusing growth on the middle layers, we can produce models that are significantly more capable at reasoning and mathematics without increasing pretraining costs.
Future work will explore whether this bias persists at the 70B+ parameter scale and investigate the slight performance trade-offs observed in memorization-heavy tasks like closed-book QA. Formalizing the mathematical relationship between stacked architectures and looped models remains a key theoretical priority.
Summary updated