In the fast-paced world of Large Language Models (LLMs), architectural innovation is often a "black box." While industry giants like DeepSeek or Qwen release powerful models, the graveyard of ideas that didn't work—and the rigorous scaling data for those that did—rarely sees the light of day. Meanwhile, academic researchers often struggle with "compute poverty," producing interesting ideas on small scales that may not hold up when training budgets explode.
ArchSpace is a new open-source initiative aiming to bridge this gap. It provides a standardized platform for recording, reviewing, and referencing LLM architecture research, effectively bringing industry-grade validation practices to the broader community.
TL;DR
ArchSpace is a public platform and validation framework that uses scaling laws to democratize LLM architecture research. By providing standardized baselines (Qwen3.5 and OLMo3) and a rigorous "scaling ladder" evaluation, it allows researchers to reliably predict the large-scale impact of architectural changes using only a fraction of the compute.
What Problem Does This Paper Address?
The authors identify a massive "visibility gap" in AI research. As shown in the paper's analysis of publication trends, academic work accounts for roughly a third of architecture papers but receives less than 10% of the total citations.
In industry, architectural choices are validated through expensive scaling sweeps that are rarely published. In academia, limited compute leads to "single-point" comparisons (e.g., comparing two 100M parameter models) which often fail to translate to 7B+ parameter models. ArchSpace provides the infrastructure to make these comparisons rigorous, reproducible, and predictive.
Methods
The core of ArchSpace is its Two-Phase Validation Framework:
- Phase I: The Scaling Ladder: Instead of a single training run, researchers test their proposed architecture across five compute budgets (from $10^{18}$ to $10^{20}$ FLOPs). By fitting scaling laws to these runs, they can predict how the model would perform at much larger scales.
- Phase II: 1B Model Training: To ensure that gains in "loss" actually translate to "capabilities," the best candidates are trained as 1B parameter models on 150B tokens, followed by a specific long-context "mid-training" phase.
Technically, the paper introduces a more sophisticated "coupled" scaling law. While the famous Chinchilla study used an additive form, ArchSpace uses a coupled exponent that better captures how model size and data quantity interact at the edges of the scaling curve. They also provide a novel way to model the impact of Learning Rate (LR) and Batch Size (BS) using hyperbolic cosine (cosh) terms, helping researchers find the optimal "recipe" for a new architecture faster.
Datasets
ArchSpace standardizes the data to ensure "apples-to-apples" comparisons:
- Pretraining: A 150B token sample from the Dolma 3 Pool.
- Long-Context Mid-training: The Dolma 3 Longmino Mix (50B tokens).
- Evaluation: The OLMES suite for general capabilities and RULER/HELMET for long-context performance.
Key Insights and Ideas
- Predictive Power: The authors demonstrate that scaling laws fitted on small models (under 1B parameters) can predict the loss of models 10x larger with an error of only ~0.32%. This allows for "failing fast" on bad ideas without spending millions on compute.
- Coupling Matters: The interaction between model size ($M$) and data ($D$) is non-trivial. The paper shows that a "coupled" scaling law fits the data significantly better than the standard additive Chinchilla form, especially when predicting performance far outside the training range.
- The Importance of Mid-Training: For long-context performance, simply having a good architecture isn't enough. The Phase II results show that a dedicated 10B token mid-training stage is crucial for models to actually utilize long context windows (up to 64K).
Key Results
- Baseline Release: The authors provide full scaling data and training recipes for Qwen3.5 and OLMo3 (hybrid architectures combining attention with Gated DeltaNet or sliding windows).
- Scaling Law Accuracy: Their fitted laws achieved a Mean Absolute Percentage Error (MAPE) of just 0.25% for Qwen3.5 and 0.27% for OLMo3 across the scaling ladder.
- Long-Context Gains: Using their recipe, Qwen3.5’s RULER score (a measure of effective context usage) jumped from 17.5 to 57.0 after mid-training.
Limitations and Future Work
While ArchSpace is a major step forward, the authors acknowledge several hurdles:
- Downstream Non-monotonicity: While "loss" scales predictably, specific task performance (like coding or math) can sometimes be "jumpy" or non-monotonic, making it harder to predict with simple power laws.
- Architecture Diversity: The current baselines are hybrids. Future work is needed to incorporate Mixture-of-Experts (MoE) and other emerging architectures into the ladder.
- Community Adoption: The success of ArchSpace depends on researchers moving away from "one-off" papers and toward contributing to this shared, living record of architectural experiments.
Summary updated