AlterNet: Adaptive-Depth Mixture-of-Experts with Learned Halting for Elastic Inference

Alexander KastiusarXiv 2026 · arXiv

TL;DR

AlterNet is a novel architecture that merges Mixture-of-Experts (MoE) with a dynamic-depth loop, allowing Large Language Models (LLMs) to adjust their computational effort on the fly. By using learned halting mechanisms and depth-aware routing, the model can "think longer" for hard problems and "exit early" for easy ones, enabling faster and cheaper inference without the need for retraining.

What Problem Does This Paper Address?

Most modern LLMs have a "fixed compute" problem: whether you ask them to solve a complex math proof or just say "hello," they use the exact same number of layers and parameters. While Mixture-of-Experts (MoE) models offer some sparsity by only activating a subset of parameters, the total number of layers (depth) remains static.

If you want a smaller, faster version of a model, you typically have to perform expensive retraining, distillation, or quantization. AlterNet aims for elastic inference—an architecture where the model itself determines the necessary compute depth for a given task, and where developers can scale that compute down after training to meet specific hardware constraints.

Methods

AlterNet replaces the standard stack of transformer layers with an iterative loop containing a pool of "shallow experts."

Figure 1: AlterNet forward pass. Input tokens are embedded and passed through a depth loop with a maximum of D iterations. At each step, a depth-aware router selects k of E shallow experts; a PonderNet halt gate accumulates per-step probabilities against a Gaussian prior.

Key technical innovations include:

  • Iterative Computation: Instead of passing through $N$ distinct layers, the hidden state cycles through a loop for a maximum of $D$ iterations.
  • Depth-Aware Routing: A router selects $k$ experts from the pool at each step. Crucially, the router receives a sinusoidal depth encoding (similar to positional encodings) that tells it which iteration it is currently in ($t$) and how much compute budget remains ($1-t$).
  • Learned Halting (PonderNet++): Each expert has a "halt head" that predicts the quality of its output. These predictions are aggregated into a "halt gate" that decides whether to stop the loop early.
  • Gaussian Prior: While previous work (like PonderNet) used a geometric distribution that biased models toward very shallow processing, AlterNet uses a Gaussian prior centered at $D/2$. This encourages the model to use a "bell curve" of compute, preventing it from getting stuck at depth 1.

Datasets

The authors evaluated AlterNet using the C4 (English) dataset. They compared their results against a baseline of OPT 350M, a standard dense transformer model.

Key Insights and Ideas

  • Sequence-Level Aggregation: Unlike many MoE models that route every token independently (which can be unstable), AlterNet uses the last valid token in a sequence as an aggregator for routing decisions. This reduces overhead and stabilizes the router.
  • The "Depth-Bootstrap Collapse": The authors identified a critical failure mode where models get "stuck" at the first iteration. Because the loss for the first step is independent of later steps, the model often learns to rely on it exclusively. They solved this using a combination of the Gaussian prior and a "curriculum release" training strategy.
  • Budget Signaling: Explicitly passing the remaining compute budget ($1-t$) to the router was found to be essential. Removing this signal significantly hurt performance in ablation studies.
  • Expert Specialization: By providing a depth signal, the authors allow the same pool of experts to specialize. An expert might learn to be a "low-level feature extractor" when called in iteration 1, but a "logical reasoner" when called in iteration 10.

Key Results

The researchers trained two 350M parameter models: Model I (fixed at 12 iterations) and Model II (adaptive).

  • Compute Savings: Model II successfully learned to halt, converging to an average of 6 iterations per forward pass—a 50% reduction in compute compared to the fixed-depth baseline.
  • Performance Trade-off: At 30k training steps, the adaptive Model II achieved a perplexity (PPL) of 54.3. While this is higher than the fixed-depth Model I (40.1 PPL at 10k steps), it demonstrates that the model can learn to function with significantly less compute.
Figure 3: Average iterations per forward pass during training. The model converges toward the Gaussian prior, saving ≈50% of iterations.

As seen in the training logs, the model's average iterations stabilized around the Gaussian prior, proving the stability of the halting mechanism.

Limitations and Future Work

The authors acknowledge that their models are currently significantly undertrained (3.9B tokens vs. the recommended 7B+ for this size) due to resource constraints. Consequently, AlterNet does not yet outperform established baselines like OPT in raw perplexity.

Future directions include: * Scaling Up: Testing the architecture on multi-billion parameter scales and deeper iteration loops. * Domain-Specific Experts: Investigating if experts can be "pre-loaded" or "dropped" based on the detected depth and domain of the input. * Refined Halting: Improving the balance between the five different loss components to close the performance gap with fixed-depth models.

Summary updated

Abstract

Most LLM architectures have fixed compute demands determined at training time. We propose a novel architecture that combines the Mixture-of-Experts architecture with shallow experts and a novel dynamic-depth system to make the extent of a forward pass dynamically adjustable. Our approach lets the model automatically determine the compute needed for a given task, paired with a system to scale it down after training without retraining when the execution environment demands faster or cheaper inference. To achieve this, we introduce an iterative computation mechanism, sequence-level depth-based routing steered by a halting mechanism within the experts, a depth encoding that lets the model track its progress in the chain, and a regularization mechanism that incentivizes the model to use allocated compute efficiently. We present initial benchmarks and discuss future research directions.
BibTeX
@misc{kastius2026alternet,
  author = {Alexander Kastius},
  title = {{AlterNet: Adaptive-Depth Mixture-of-Experts with Learned Halting for Elastic Inference}},
  year = {2026},
  eprint = {2611.08741},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08741},
}
Figure