LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling

Jian Yang, Shawn Guo, Wei Zhang, Tianyu Zheng, Yaxin Du, Haau-Sing Li, Jiajun Wu, Yue Song, Yan Xing, Qingsong Cai, Zelong Huang, Chuan Hao, Ran Tao, Xianglong Liu, Wayne Xin Zhao, Mingjie Tang, Weifeng Lv, Ming Zhou, Bryan DaiarXiv 2026 · arXiv

TL;DR

Looped Transformers scale a model's depth by passing data through a shared block multiple times, but standard implementations suffer from high latency and memory overhead. Parallel Loop Transformers (PLT) solve this efficiency bottleneck, but adding more loops surprisingly degrades performance after a certain point. This paper introduces LoopCoder-v2 to demonstrate that a strict two-loop limit is optimal, driven by a fundamental "gain-cost" trade-off between representational refinement and positional mismatch.

What Problem Does This Paper Address?

Looped large language models (LLMs) offer an elegant way to scale test-time compute without blowing up parameter counts. By repeatedly applying a single shared Transformer block, the model can perform multiple rounds of latent-space computation—essentially doing "implicit reasoning" before emitting a token.

However, standard sequential looping forces inference latency and KV-cache memory to scale linearly with the number of loops, making it impractical for deployment. While Parallel Loop Transformers (PLT) alleviate these costs by executing loops concurrently, they introduce a new dilemma: How many loops should we actually use? Prior to this work, the internal mechanics dictating the optimal loop count were poorly understood, leaving engineers to rely on expensive, brute-force benchmarking.

Table 1. Sequential looping versus PLT, where 𝐶block is the cost of one pass through the shared block. PLT keeps both latency and memory independent of the loop count 𝑅.

Methods

The authors investigate loop-count selection by training LoopCoder-v2, a family of 7B-parameter PLT models, from scratch. The PLT architecture relies on two key efficiency mechanisms:

  1. Cross-Loop Position Offsets (CLP): This mechanism right-shifts the previous loop’s hidden states by one token position. This breaks sequential inter-loop dependencies, allowing the model to process multiple loops in parallel within a single forward pass.
  2. Shared-KV Gated Sliding-Window Attention (G-SWA): To bound memory usage, the model freezes and shares the KV cache from the first loop across all subsequent iterations, fusing it dynamically with local sliding-window attention.

To diagnose what happens inside the model as loops accumulate, the authors develop a suite of "microscopic" interpretability lenses. They track hidden-state dynamics (step size, angular alignment, and effective rank), attention heat-map evolution, and output-distribution shifts across each individual loop.

Datasets

LoopCoder-v2 was trained from scratch on a massive internal deduplicated mixture of 18 trillion tokens, featuring a balanced 1:1 text-to-code ratio spanning over 100 programming languages (with Java, Python, and JavaScript leading the share).

For downstream evaluation, the models underwent matched instruction tuning on 6 million examples and were evaluated on a comprehensive benchmark suite covering:

  • Code Generation & Reasoning: HumanEval+, MultiPL-E, LiveCodeBench (LCB), BigCodeBench (BCB).
  • Agentic Software Engineering: SWE-bench Verified, Multi-SWE.
  • Tool Use & Environment Interaction: Terminal-Bench, Mind2Web, BFCL.

Key Insights and Ideas

  • The Non-Monotonic Trap: Performance does not scale linearly with more loops. It peaks sharply at exactly two loops ($R=2$) and severely regresses when a third or fourth loop is added.
  • The Gain-Cost Scissors: Why does performance saturate at two loops? The authors reveal a "scissors" effect. Loop 2 provides highly productive refinement—maximizing representational diversity and shifting output distributions toward the correct answers. However, the CLP mechanism introduces a structural cost: because it shifts tokens, it creates a positional mismatch. While the refinement gain collapses after loop 2, this positional mismatch tax remains fixed and high across all subsequent loops, quickly dominating and degrading performance.
The Gain-Cost Scissors effect in PLT
  • Latent and Explicit CoT are Partners, Not Rivals: Internal latent looping ($R=2$) and explicit token-level Chain-of-Thought (generating "thinking" tokens) are entirely complementary. When combined, they act super-additively: explicit CoT decomposes a problem textually, while latent loops refine the underlying representations at each step.

Key Results

  • Massive Gains at Two Loops: LoopCoder-v2 ($R=2$) achieved sweeping improvements over the non-looped baseline ($R=1$). Most notably, on SWE-bench Verified, performance jumped from 43.0% to 64.4%, allowing this 7B model to rival massive 480B-scale open models.
  • Severe Regression at Three Loops: Adding a third loop ($R=3$) caused a catastrophic drop on SWE-bench Verified, plummeting down to 27.6%.
  • Super-Additive Thinking Boost: Pairing the optimal $R=2$ model with an explicit reasoning trace ("thinking" mode) yielded a massive +26.9 point boost on LiveCodeBench compared to the latent-loop-only baseline.

Limitations and Future Work

The primary limitation identified is the rigid, fixed cost imposed by the CLP positional mismatch. Because this structural tax does not decrease as the model loops deeper, it fundamentally caps the utility of parallel latent reasoning at two loops.

The authors note that future work should focus on developing adaptive offset mechanisms or dynamic loop allocation strategies that can bypass this fixed cost. Additionally, exploring how parameter-tied latent recurrence can be more deeply integrated with explicit token-level reasoning remains an open and promising frontier.

Summary updated

Abstract

Looped Transformers scale latent computation by repeatedly applying shared blocks, but sequential looping increases latency and KV-cache memory with the loop count. Parallel loop Transformers (PLT) alleviate this cost through cross-loop position offsets (CLP) and shared-KV gated sliding-window attention, making loop count a practical design choice. We therefore study PLT loop-count selection through a gain--cost view: an extra loop may refine representations, but CLP also introduces a positional mismatch at each loop boundary. We instantiate this study by training LoopCoder-v2, a family of 7B PLT coders with different loop counts, from scratch on 18T tokens, followed by matched instruction tuning and evaluation. Empirically, the two-loop variant delivers broad gains over the non-looped baseline across code generation, code reasoning, agentic software engineering, and tool-use benchmarks, improving SWE-bench Verified from 43.0 to 64.4 points and Multi-SWE from 14.0 to 31.0 points. In contrast, variants with three or more loops regress, revealing a strongly non-monotonic loop-count effect. Our diagnostics show that loop 2 provides the main productive refinement, while later loops yield diminishing, oscillatory updates and reduced representational diversity. Because the CLP-induced mismatch remains roughly fixed as refinement gains shrink, the offset cost increasingly dominates. This gain--cost trade-off explains PLT's saturation at two loops and provides diagnostics for loop-count selection.
BibTeX
@misc{yang2026loopcoder,
  author = {Jian Yang and Shawn Guo and Wei Zhang and Tianyu Zheng and Yaxin Du and Haau-Sing Li and Jiajun Wu and Yue Song and Yan Xing and Qingsong Cai and Zelong Huang and Chuan Hao and Ran Tao and Xianglong Liu and Wayne Xin Zhao and Mingjie Tang and Weifeng Lv and Ming Zhou and Bryan Dai},
  title = {{LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling}},
  year = {2026},
  eprint = {2606.18023},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.18023},
}

Loading related content...

Figure