Folding Language Models into Themselves

Ziyang Zhang, Yubin Jing, Yuyao Li, Haofan Wang, Yichen GongarXiv 2026 · arXiv

TL;DR

Researchers have introduced Concertina, a novel method that compresses large language models by "folding" adjacent layers into a single shared block that runs multiple times. By using low-rank corrections and per-token gating, Concertina recovers up to 90% of a teacher model's reasoning capabilities with 1,000x less data than training a compact model from scratch.

What Problem Does This Paper Address?

The standard way to get a small, efficient language model is through knowledge distillation: you take a massive "teacher" model and train a smaller "student" model to mimic it. However, most students start from a random initialization, essentially throwing away the vast amount of knowledge already encoded in the teacher's weights.

Because of this "blank slate" start, even tiny models today require trillions of tokens to reach high performance. The real cost of a compact model isn't its size—it's the massive compute and data needed to rebuild it. The authors of Concertina ask: instead of starting over, can we just fold the teacher's existing knowledge into a smaller architecture?

Methods

Concertina introduces a structural "folding" technique combined with dynamic adaptation.

  1. Adjacent Folding: The model splits the teacher’s $L$ layers into groups of $N$ adjacent layers. Each group is replaced by one shared block that is executed $N$ times in a loop.
  2. SVD-Initialized Step-LoRA: A single shared weight can't perfectly represent $N$ different layers. To fix this, the authors add a low-rank correction (like LoRA) for each step in the loop. Crucially, these aren't initialized to zero; they use Singular Value Decomposition (SVD) to capture the actual difference between the shared weight and the original teacher layer it replaces.
  3. Input-Conditioned Step Gate: Not all tokens need the same "correction." Concertina uses a per-token gate that looks at the current hidden state to decide how to mix these corrections at each step. This allows the model to dynamically adjust its behavior for every single word it processes.
Figure 1. Overview of Concertina. Left: the layers of the pretrained decoder are split into groups G1, . . . , GL/N of N adjacent layers. Middle: each group is replaced by one shared block that is executed N times, and red dots mark the folded linear maps. Right: each folded linear map adds low-rank corrections BkAk to the shared weight W0, mixed by gate weights gk that depend on the token x and the loop step t. Residual connections are omitted for clarity.

Datasets

The authors distilled their model using a 22-billion token recipe, which is tiny compared to the trillions used for models like Llama or Qwen. The mix was carefully balanced:

  • 35% Mathematical text and solutions (synthetic and web).
  • 55% Knowledge-oriented data (multiple-choice questions, STEM articles, academic papers).
  • 10% General web text.

They evaluated performance across eight major benchmarks, including GSM8K and MATH for reasoning, MMLU for broad knowledge, and a "Base-7" average for commonsense tasks like HellaSwag and WinoGrande.

Key Insights and Ideas

  • Adjacent is Better than Cyclic: Previous "layer sharing" methods often tied distant layers (e.g., layer 1 with layer 19). Concertina proves that adjacent layers are much more similar in their function (see Figure 4c), making them far easier to "fold" together without losing information.
  • Depth vs. Parameters: The paper reveals a fascinating split in model capabilities. Depth seems to support "procedural" skills like multi-step reasoning, while parameters (width/unique weights) support factual recall. By keeping the depth (looping) but sharing parameters, Concertina preserves reasoning much better than simply cutting layers out (pruning).
  • Data Quality > Quantity: For an "inherited" student that already has the teacher's weights, the mixture of data matters more than the volume. Targeted data repairs the specific "read-out" paths the model lost during folding.
Figure 4. Controlled comparison of the RRT design choices with ours on Qwen3-4B, all trained with the same distillation recipe. (a) Base-7 accuracy of cycle tying as used by RRT and of our adjacent tying at equal size (2.27B) and equal token budget. (b) Base-7 accuracy of the static depth-wise LoRA used by RRT and of our input-conditioned gate on the main configuration. (c) Mean cosine similarity between the residual updates of the two layers that each topology ties together, measured on the teacher.

Key Results

The researchers tested Concertina by folding a Qwen3-4B teacher into a 2.27B student.

  • Performance Recovery: The folded model retained 90% of the teacher's performance on commonsense and math reasoning.
  • Efficiency: It outperformed Qwen3-1.7B, a model of similar size that was trained on over 1,000 times more data.
  • Folding vs. Pruning: Compared to a "truncated" dense model of the same size (which just keeps the first 19 layers), Concertina was 41 times better on GSM8K and 11 times better on MATH. This proves that keeping the "looping" depth is vital for logic.
Figure 2. Scores of the main model on the eight benchmarks during training. Each line follows one benchmark at every 2.2B distillation tokens. Dashed lines mark the scores of the Qwen3-4B teacher in the color of each benchmark.

Limitations and Future Work

While Concertina is a breakthrough for memory-constrained devices (it significantly reduces the number of parameters stored), it does not reduce the total number of FLOPs (floating-point operations) or the size of the KV cache. Because the model still runs $N$ loop steps, the computational cost remains similar to the original teacher.

Future work could explore how to combine folding with width pruning to reduce FLOPs, or testing the method on even larger model families (100B+ parameters) to see if the "redundancy" of adjacent layers increases with scale.

Summary updated

Abstract

Knowledge distillation usually trains a compact student from scratch, which needs a large amount of data even though the teacher's weights already encode most of what the student has to learn. In this work, we explore building the student directly from the teacher's weights. We propose Concertina, which folds every group of N adjacent teacher layers into one shared block that is executed N times. Each loop step carries a low-rank correction initialized from the layer it replaces, and a per-token gate mixes these corrections, to let every token recover the behavior of the original layers from a single shared block. We further design a distillation data recipe that balances general, mathematical, academic and targeted multiple-choice data, to recover all capabilities of the teacher together rather than one at the expense of another. On Qwen3-4B, a 2.27B Concertina model distilled on 22B tokens retains about 90% of the teacher on commonsense and mathematical reasoning and 84% on average across eight benchmarks. It outperforms same-size dense and layer-sharing baselines, as well as Qwen3-1.7B, which is trained on over a thousand times more data. These results suggest that a compact model need not be retrained at scale, since folding the teacher into itself and distilling it on the right data recovers most of its capabilities. Code is available at https://github.com/myboker/concertina.
BibTeX
@misc{zhang2026folding,
  author = {Ziyang Zhang and Yubin Jing and Yuyao Li and Haofan Wang and Yichen Gong},
  title = {{Folding Language Models into Themselves}},
  year = {2026},
  eprint = {2611.18113},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18113},
}
Figure