Ruyi2 Technical Report

Huan Song, Shuyu Tian, Junyi Hao, Minxiu Xu, Hongjun An, Yiliang Song, Jiawei Shao, Xuelong LiarXiv 2026 · arXiv

TL;DR

Ruyi2 introduces a highly efficient "Familial Model" framework that embeds multiple deployable sub-models (1.7B, 8B, and 14B) within a single shared Transformer backbone. By making this early-exit architecture fully compatible with large-scale 3D parallel training, the authors achieve a 2–3× training speedup over previous methods while significantly outperforming same-sized baselines like Qwen3.

What Problem Does This Paper Address?

Deploying Large Language Models (LLMs) in production is notoriously expensive and latency-heavy. While "adaptive computing"—specifically early-exit architectures—offers a solution by allowing simpler inputs to exit the network early, scaling these models has proven difficult.

Prior early-exit methods suffer from two major flaws:

  1. Optimization Complexity: Training multiple exits simultaneously causes gradient interference across shared layers, destabilizing the learning process.
  2. Distributed Training Incompatibility: Standard early-exit setups do not natively play well with the 3D parallelism (data, tensor, and pipeline parallelism) required to train billion-parameter models.

Ruyi2 bridges this gap by providing a stable, scalable framework built on Megatron-LM, realizing a true "Train Once, Deploy Many" paradigm.

Methods

The core of Ruyi2 is its Familial Model design, which shares a single backbone (built on Qwen3-14B-Base) across three nested sub-models: Ruyi2-1.7B (layers 1–3), Ruyi2-8B (layers 1–22), and Ruyi2-14B (all 40 layers).

Overview of the Staged Capability Improvement Pipeline

To train this architecture efficiently, the authors introduce several key innovations:

1. Joint Optimization Framework

Instead of training separate models, all branches are optimized simultaneously. The total loss is a weighted sum of the cross-entropy losses from the main backbone and the auxiliary exits: $$\mathcal{L}_{\text{total}} = \sum_{k=1}^{K} \lambda_k \mathcal{L}_{\text{branch}\_k}$$ Crucially, the weighting coefficient $\lambda_k$ is time-dependent. It provides deep supervision to lower layers early in training to accelerate convergence, then progressively decays to prevent auxiliary objectives from interfering with the main backbone's final optimization.

2. The DaE (Expand-then-Decompose) Framework

To boost the performance of the edge-focused 1.7B variant without starting from scratch, the authors developed the DaE framework:

Two stages of DaE
  • Stabilized Block Expansion (SBE): They append $N=3$ additional Transformer blocks to the 1.7B exit. To maintain stability, they use a zero-residual initialization strategy (setting final projection matrices to zero), forcing the new blocks to initially act as an exact identity mapping.
  • Randomized Internal Initialization: Instead of copying weights from adjacent layers (layer cloning), the internal weights are initialized randomly. This breaks "optimization inertia" and prevents representation redundancy.
  • Post-Training Low-Rank Decomposition: After alignment, Singular Value Decomposition (SVD) compresses these newly introduced blocks, slashing the incremental parameter count by 40% before deployment.

Datasets

  • Pre-training: The model was continually pre-trained on an 800-billion-token proprietary corpus focusing on high-density, high-quality data spanning multilingual web text, scientific literature, code, and mathematics.
  • Instruction Tuning (SFT): Joint instruction fine-tuning was performed using 4 million high-quality examples, split into a diversity-focused stage and a domain-specialization stage (heavily weighted toward math and code).
  • Reinforcement Learning: Alignment was further refined using Group Relative Policy Optimization (GRPO) on filtered mathematical codes and a validation set for reward modeling.
  • Evaluation Benchmarks: The models were evaluated across standard benchmarks including MMLU, CMMLU, BBH, ARC-c, HellaSwag, IFEval, HumanEval, MATH, and GSM8K.

Key Insights and Ideas

  • Symmetry Breaking via Randomization: The authors discovered that standard layer cloning introduces severe optimization inertia in small-scale models. By using randomized internal initialization combined with zero-residual paths, the new layers are forced to learn novel, complementary representations rather than simply replicating existing feature extraction patterns.
  • Adherence to Scaling Laws: A major anxiety with parameter-sharing architectures is that the sharing mechanism might disrupt the backbone's scaling behavior. Ruyi2 proves that with carefully scheduled multi-exit loss aggregation, the underlying transformer backbone continues to follow predictable scaling laws.

Key Results

Ruyi2 achieves substantial performance gains over standard models of equivalent size:

  • Unmatched Knowledge Density at 1.7B: Ruyi2-1.7B achieved an MMLU score of 62.77, drastically outperforming Qwen3-1.7B (39.31).
  • Dominance at 8B and 14B: Ruyi2-8B scored 79.68 on MMLU and 92.19 on GSM8K (compared to Qwen3-8B's 48.64 and 85.37). The flagship Ruyi2-14B model led the pack with an MMLU of 81.84 and a GSM8K score of 94.24.
  • Efficient Edge Compression: The DaE framework successfully reduced the incremental parameters of the expanded edge model by 40% while keeping the performance drop under 2%.

Limitations and Future Work

Despite its success, the authors note that the 1.7B edge variant still faces difficulties in achieving the same reasoning depth as larger models on complex logic tasks, occasionally suffering from hallucinations. Furthermore, open challenges remain regarding long-context processing and the elimination of generative artifacts within early-exit frameworks.

Future iterations aim to explore extreme sparsification techniques, more sophisticated dynamic exit policies, and moving beyond text-centric paradigms into multimodal integration.

Summary updated

Abstract

Large Language Models (LLMs) face significant challenges regarding deployment costs and latency, necessitating adaptive computing strategies. Building upon the AI Flow framework, we introduce Ruyi2 as an evolution of our adaptive model series designed for efficient variable-depth computation. While early-exit architectures offer a viable efficiency-performance balance, the Ruyi model and existing methods often struggle with optimization complexity and compatibility with large-scale distributed training. To bridge this gap, Ruyi2 introduces a stable "Familial Model" based on Megatron-LM. By using 3D parallel training, it achieves a 2-3 times speedup over Ruyi, while performing comparably to same-sized Qwen3 models. These results confirm that family-based parameter sharing is a highly effective strategy, establishing a new "Train Once, Deploy Many" paradigm and providing a key reference for balancing architectural efficiency with high-performance capabilities.
BibTeX
@misc{song2026ruyi,
  author = {Huan Song and Shuyu Tian and Junyi Hao and Minxiu Xu and Hongjun An and Yiliang Song and Jiawei Shao and Xuelong Li},
  title = {{Ruyi2 Technical Report}},
  year = {2026},
  eprint = {2602.22543},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.22543},
}

Loading related content...

Figure