ArchSpace: Recording, Reviewing, and Referencing LLM Architecture Research

Jie Wang, Tong Zhu, Jiahao Kuang, Zhe Xu, Mowen Ruan, Yunkai Chai, Yuliang Liu, Dehong Zheng, Rubin Wei, Yu Cheng, Bowen Zhou, Tao Ji, Zhouhan Lin, Qi ZhangarXiv 2026 · arXiv

In the fast-paced world of Large Language Models (LLMs), architectural innovation is often a "black box." While industry giants like DeepSeek or Qwen release powerful models, the graveyard of ideas that didn't work—and the rigorous scaling data for those that did—rarely sees the light of day. Meanwhile, academic researchers often struggle with "compute poverty," producing interesting ideas on small scales that may not hold up when training budgets explode.

ArchSpace is a new open-source initiative aiming to bridge this gap. It provides a standardized platform for recording, reviewing, and referencing LLM architecture research, effectively bringing industry-grade validation practices to the broader community.

TL;DR

ArchSpace is a public platform and validation framework that uses scaling laws to democratize LLM architecture research. By providing standardized baselines (Qwen3.5 and OLMo3) and a rigorous "scaling ladder" evaluation, it allows researchers to reliably predict the large-scale impact of architectural changes using only a fraction of the compute.

What Problem Does This Paper Address?

The authors identify a massive "visibility gap" in AI research. As shown in the paper's analysis of publication trends, academic work accounts for roughly a third of architecture papers but receives less than 10% of the total citations.

Academic research produces many papers but lacks the citation impact of industry-led work, often due to inconsistent baselines and lack of large-scale validation.

In industry, architectural choices are validated through expensive scaling sweeps that are rarely published. In academia, limited compute leads to "single-point" comparisons (e.g., comparing two 100M parameter models) which often fail to translate to 7B+ parameter models. ArchSpace provides the infrastructure to make these comparisons rigorous, reproducible, and predictive.

Methods

The core of ArchSpace is its Two-Phase Validation Framework:

  1. Phase I: The Scaling Ladder: Instead of a single training run, researchers test their proposed architecture across five compute budgets (from $10^{18}$ to $10^{20}$ FLOPs). By fitting scaling laws to these runs, they can predict how the model would perform at much larger scales.
  2. Phase II: 1B Model Training: To ensure that gains in "loss" actually translate to "capabilities," the best candidates are trained as 1B parameter models on 150B tokens, followed by a specific long-context "mid-training" phase.
Figure 1: Academic architecture research receives a disproportionately small citation share, motivating ArchSpace’s goal of broader visibility and recognition.

Technically, the paper introduces a more sophisticated "coupled" scaling law. While the famous Chinchilla study used an additive form, ArchSpace uses a coupled exponent that better captures how model size and data quantity interact at the edges of the scaling curve. They also provide a novel way to model the impact of Learning Rate (LR) and Batch Size (BS) using hyperbolic cosine (cosh) terms, helping researchers find the optimal "recipe" for a new architecture faster.

Datasets

ArchSpace standardizes the data to ensure "apples-to-apples" comparisons:

  • Pretraining: A 150B token sample from the Dolma 3 Pool.
  • Long-Context Mid-training: The Dolma 3 Longmino Mix (50B tokens).
  • Evaluation: The OLMES suite for general capabilities and RULER/HELMET for long-context performance.

Key Insights and Ideas

  • Predictive Power: The authors demonstrate that scaling laws fitted on small models (under 1B parameters) can predict the loss of models 10x larger with an error of only ~0.32%. This allows for "failing fast" on bad ideas without spending millions on compute.
  • Coupling Matters: The interaction between model size ($M$) and data ($D$) is non-trivial. The paper shows that a "coupled" scaling law fits the data significantly better than the standard additive Chinchilla form, especially when predicting performance far outside the training range.
  • The Importance of Mid-Training: For long-context performance, simply having a good architecture isn't enough. The Phase II results show that a dedicated 10B token mid-training stage is crucial for models to actually utilize long context windows (up to 64K).
Phase II focuses on training 1B models to confirm that architectural gains persist through long-context mid-training and downstream evaluations.

Key Results

  • Baseline Release: The authors provide full scaling data and training recipes for Qwen3.5 and OLMo3 (hybrid architectures combining attention with Gated DeltaNet or sliding windows).
  • Scaling Law Accuracy: Their fitted laws achieved a Mean Absolute Percentage Error (MAPE) of just 0.25% for Qwen3.5 and 0.27% for OLMo3 across the scaling ladder.
  • Long-Context Gains: Using their recipe, Qwen3.5’s RULER score (a measure of effective context usage) jumped from 17.5 to 57.0 after mid-training.

Limitations and Future Work

While ArchSpace is a major step forward, the authors acknowledge several hurdles:

  • Downstream Non-monotonicity: While "loss" scales predictably, specific task performance (like coding or math) can sometimes be "jumpy" or non-monotonic, making it harder to predict with simple power laws.
  • Architecture Diversity: The current baselines are hybrids. Future work is needed to incorporate Mixture-of-Experts (MoE) and other emerging architectures into the ladder.
  • Community Adoption: The success of ArchSpace depends on researchers moving away from "one-off" papers and toward contributing to this shared, living record of architectural experiments.

Summary updated

Abstract

Model architecture shapes the scaling behavior, training efficiency, and inference cost of LLMs, making it a central focus of both industry and academia. In industry, teams such as those behind DeepSeek, Qwen, and Kimi routinely introduce architectural innovations, yet public reports rarely document which ideas succeeded, which were abandoned, and why. In academia, limited compute and inconsistent baselines often restrict evaluations to small-scale, single-point comparisons, making it difficult to establish whether improvements hold at larger scales or translate into broader impact. ArchSpace addresses both gaps as a public platform for Recording, Reviewing, and Referencing Architecture Research: each proposal is recorded with its experiments and reasoning, reviewed by the community, and referenced by later work. To evaluate ideas before costly large-scale training, ArchSpace leverages scaling laws to predict model loss at larger scales from smaller experiments, a practice widely adopted in industry. We provide initial references that make the platform usable: Qwen3.5 and OLMo3 baselines, fitted scaling laws, and a reusable training recipe. Ideas validated through this review can in turn become new baselines, and ArchSpace thereby aims to broaden the range of ideas that shape the next generation of models.
BibTeX
@misc{wang2026archspace,
  author = {Jie Wang and Tong Zhu and Jiahao Kuang and Zhe Xu and Mowen Ruan and Yunkai Chai and Yuliang Liu and Dehong Zheng and Rubin Wei and Yu Cheng and Bowen Zhou and Tao Ji and Zhouhan Lin and Qi Zhang},
  title = {{ArchSpace: Recording, Reviewing, and Referencing LLM Architecture Research}},
  year = {2026},
  eprint = {2611.17332},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17332},
}
Figure