Dory1.0-4B: A Looped Normalized-GPT

Maxim Fishman, Simeng Sun, Shantanu Acharya, Faisal Ladhak, Brian Chmiel, Santiago Akle Serano, Ron Banner, Boris Ginsburg, Krishna C. PuvvadaarXiv 2026 · arXiv

TL;DR

Dory1.0-4B is a 4-billion-parameter language model that uses "elastic depth"—a recurrent architecture that allows it to trade inference compute for accuracy by looping through its layers multiple times. By spending more time "thinking" in latent space rather than just generating more tokens, this 4B model outperforms much larger competitors (up to 26B parameters) and solves complex agentic tasks with significantly fewer output tokens.

What Problem Does This Paper Address?

Traditionally, scaling a Large Language Model (LLM) means increasing its parameter count or generating longer "Chain-of-Thought" reasoning traces. Both approaches are expensive: more parameters require more memory, and longer outputs increase latency and serving costs (KV-cache footprint).

The authors address this by exploring looped (latent-recurrent) language models. Instead of adding more unique layers, they repeatedly apply a shared block of layers. This allows the model to scale its "test-time compute" per token without increasing the model's size or the length of the generated text.

Methods

Dory1.0-4B is built on the normalized Transformer (nGPT) backbone, which uses L2 normalization to keep activations stable—a crucial feature for recurrent architectures where errors can easily compound.

Architecture

The model consists of 72 physical layers organized into three sections:

  1. Input Block (16 layers): Processes the initial token embedding.
  2. Recurrent Block (40 layers): The "engine" that can be run $k$ times (where $k=1$ to 4).
  3. Output Block (16 layers): Prepares the final state for the language model head.
The looped nGPT architecture showing how the recurrent block is applied $k$ times.

Training Pipeline

  • Massive Pretraining: Trained on 6 trillion tokens. The first 4.5T used NVFP4 (4-bit) for a 20% speedup, followed by a BF16 continuation to introduce looping and extend the context window to 256K tokens.
  • RLVR with a Twist: The model underwent Reinforcement Learning with Verifiable Rewards (RLVR). The key innovation here is a difficulty-aware loop curriculum: during training, harder prompts were dynamically routed to deeper loops ($k=4$), while easier ones stayed at $k=1$.

Inference Optimizations

  • Last-Loop KV Caching: To save memory, the model only stores the Key-Value (KV) cache from the final loop. This keeps the memory footprint identical to a non-looped model regardless of how many times it "thinks."
  • Self-Speculative Decoding: The model uses its own $k=1$ setting to quickly draft tokens and its $k=4$ setting to verify them, accelerating inference by up to 1.53x.

Datasets

Dory was trained on the Nemotron 3 Ultra and Nemotron 3.5 Super data blends. It was evaluated across 12 diverse benchmarks, including:

  • Agentic/Coding: SciCode, SWE-Bench Verified, Terminal-Bench v2.1, $\tau^3$-Banking, and GDPval-AA v2.
  • Reasoning: GPQA Diamond and Humanity’s Last Exam (HLE).
  • Long Context: AA-LCR and GDP.pdf.
  • General Knowledge: MMLU-Pro and AA-Omniscience.

Key Insights and Ideas

  • Thinking vs. Talking: Increasing the loop count ($k$) improves accuracy significantly, especially in coding and agentic tasks. Interestingly, as the model "thinks" more in latent space (higher $k$), it actually becomes more token-efficient, solving tasks in 1.3–2.5x fewer tokens because it makes fewer mistakes and requires fewer turns.
  • Elasticity is Free: A single checkpoint can be served at any loop count. A developer can choose $k=1$ for low-latency chat or $k=4$ for complex software engineering tasks without swapping models.
  • The "Aha" of Last-Loop Caching: The authors found that you don't need to cache every intermediate state of the loop. Overwriting the cache at each iteration retains most accuracy gains while keeping memory costs low.

Key Results

Dory1.0-4B punches way above its weight class:

  • Performance Scaling: On an average of 12 benchmarks, the score jumps from 35.2 ($k=1$) to 40.6 ($k=4$).
  • Giant Killing: At $k=4$, Dory outperforms Gemma 4 26B-A4B and Qwen3.5-9B, and comes within 1.7 points of Qwen3.6-35B-A3B—a model nearly 9x its size.
  • Efficiency: Quantized versions (NVFP4) provide up to a 2.44x speedup over standard BF16 inference.
Average score increases with loop count $k$, while output tokens per attempt decrease.

Limitations and Future Work

  • Diminishing Returns: Performance tends to saturate after four loops ($k=4$). Simply looping infinitely doesn't lead to infinite intelligence.
  • Task Specificity: While agentic and reasoning tasks benefit immensely from looping, factual knowledge tasks (like AA-Omniscience) show almost no improvement, suggesting that "thinking harder" doesn't help if the model simply hasn't memorized the fact.
  • Future Directions: The authors suggest exploring even deeper loops and more sophisticated routing mechanisms to decide exactly how much "thought" each specific token requires.

Summary updated

Abstract

We introduce Dory1.0-4B, a 4-billion-parameter looped (latent-recurrent) language model with elastic depth: a single checkpoint serves different inference compute budgets. Dory1.0-4B is built on a normalized Transformer (nGPT) backbone with interleaved sliding-window and full-attention layers, organized into an input block, a shared recurrent block, and an output block. At inference, the recurrent block can be applied a variable number of times. We pre-trained Dory1.0-4B on 6 trillion tokens. The first 4.5 trillion tokens were trained in NVFP4, and additional pre-training then extended the context length to 256K tokens. Next, we post-trained the model using Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR), with a difficulty-aware loop curriculum that routes harder prompts to deeper loops. Increasing the loop count improves accuracy, with the largest gains on coding, agentic, and long-context benchmarks. With two or more loops, Dory1.0-4B outperforms larger models such as Qwen3.5-9B, Gemma 4 12B, and Gemma 4 26B-A4B on average across 12 benchmarks. Looping also makes Dory1.0-4B more token-efficient on agentic benchmarks: with four loops, it generates 1.3-2.5x fewer tokens than with one loop, as the extra computation is spent in latent space rather than on longer outputs. To reduce memory during looping, Dory1.0-4B caches keys and values only from the last loop, keeping its KV-cache footprint equal to that of a non-looped model at any loop count while retaining most of the accuracy gains. We open-source the BF16, MXFP8, NVFP4, and GGUF quantized checkpoints.
BibTeX
@misc{fishman2026dory,
  author = {Maxim Fishman and Simeng Sun and Shantanu Acharya and Faisal Ladhak and Brian Chmiel and Santiago Akle Serano and Ron Banner and Boris Ginsburg and Krishna C. Puvvada},
  title = {{Dory1.0-4B: A Looped Normalized-GPT}},
  year = {2026},
  eprint = {2611.03998},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03998},
}
Figure