Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet

Joel Lidin, Amir Sarfi, Erfan Miahi, Quentin Anthony, Shivam Chauhan, Evangelos Pappas, Benjamin Therien, Eugene Belilovsky, Samuel Dare

Abstract

Recently, there has been increased interest in globally distributed training, which has the promise to both reduce training costs and democratize participation in building large-scale foundation models. However, existing models trained in a globally distributed manner are relatively small in scale and have only been trained with whitelisted participants. Therefore, they do not yet realize the full promise of democratized participation. In this report, we describe Covenant-72B, an LLM produced by the largest collaborative globally distributed pre-training run (in terms of both compute and model scale), which simultaneously allowed open, permissionless participation supported by a live blockchain protocol. We utilized a state-of-the-art communication-efficient optimizer, SparseLoCo, supporting dynamic participation with peers joining and leaving freely. Our model, pre-trained on approximately 1.1T tokens, performs competitively with fully centralized models pre-trained on similar or higher compute budgets, demonstrating that fully democratized, non-whitelisted participation is not only feasible, but can be achieved at unprecedented scale for a globally distributed pre-training run.

Introduction

Large-scale AI training has traditionally been a centralized endeavor, requiring thousands of GPUs in a single data center connected by ultra-low-latency interconnects. This creates a significant barrier to entry, limiting pre-training to a handful of massive technology companies. Decentralized training—pooling GPUs from disparate locations—offers a path toward democratizing AI, but faces two primary hurdles: the bandwidth bottleneck of the commodity internet and the "trust gap" inherent in permissionless systems where anyone can join.

Covenant-72B addresses these challenges, representing the first 70B+ parameter model pre-trained on a globally distributed, permissionless network of trustless peers.

Methodology

The success of Covenant-72B relies on two core innovations: the SparseLoCo optimizer and the Gauntlet validation mechanism.

SparseLoCo Optimizer

SparseLoCo is a local-update distributed optimizer designed for high-latency environments. Instead of synchronizing at every step, peers perform 30 local "inner steps." During synchronization, SparseLoCo employs Top-k sparsification, 2-bit quantization, and error feedback to achieve a 146x communication compression ratio. This allows the 72B model to be trained over standard internet links.

System architecture showing the SparseLoCo optimizer and memory offloading strategy across globally distributed peers.

To maximize GPU memory efficiency, the system utilizes a dynamic FSDP (Fully Sharded Data Parallel) strategy. As shown in the architecture diagram, optimizer states are offloaded to the CPU during the computation phase and swapped back during the communication phase, freeing up GPU VRAM for model parameters and gradients.

Gauntlet Validation

To maintain model integrity in a permissionless environment, the system uses the Gauntlet mechanism on the Bittensor blockchain. A validator scores peer submissions based on LossScore, which measures actual loss improvement on specific data batches. This prevents "lazy" or malicious actors from poisoning the model without requiring a whitelist.

Training and Data

The model was pre-trained on approximately 1.1T tokens from the DCLM dataset. Following pre-training, the model underwent an annealing phase (14.2B tokens) using a curated mix of instruction, synthetic web, code, and math data. Finally, the model was fine-tuned (SFT) on 14.8B tokens focused on conversational, agentic, and STEM datasets to produce the chat version.

Results and Efficiency

Covenant-72B demonstrates that decentralized training can match the performance of centralized counterparts. On zero-shot benchmarks like ARC-Challenge and MMLU, it outperformed LLaMA-2-70B and LLM360 K2 Diamond. Notably, it achieved an MMLU score of 67.1, despite being trained on fewer tokens than its competitors.

Comparison of compute and communication/idle time between INTELLECT-1 and Covenant-72B, showing significantly reduced idle time for the larger model.

A key highlight is the system's efficiency. Despite the 7.2x increase in model size compared to previous decentralized runs (e.g., INTELLECT-1), Covenant-72B maintained a compute utilization of approximately 94.5%. As illustrated in the timing comparison, the idle time per sync round was reduced to just 70 seconds, proving that massive scale does not necessarily lead to prohibitive communication overhead in a decentralized setting.

Conclusion

Covenant-72B proves that permissionless, globally distributed pre-training is feasible at the 70B+ parameter scale. While current limitations include a relatively short context window (8k) and a requirement for homogeneous hardware (8x B200 nodes), the project demonstrates that robust validation and efficient optimization can break the centralization monopoly in AI. Future work will focus on extending context length and supporting heterogeneous compute resources.

Summary updated

Abstract

Recently, there has been increased interest in globally distributed training, which has the promise to both reduce training costs and democratize participation in building large-scale foundation models. However, existing models trained in a globally distributed manner are relatively small in scale and have only been trained with whitelisted participants. Therefore, they do not yet realize the full promise of democratized participation. In this report, we describe Covenant-72B, an LLM produced by the largest collaborative globally distributed pre-training run (in terms of both compute and model scale), which simultaneously allowed open, permissionless participation supported by a live blockchain protocol. We utilized a state-of-the-art communication-efficient optimizer, SparseLoCo, supporting dynamic participation with peers joining and leaving freely. Our model, pre-trained on approximately 1.1T tokens, performs competitively with fully centralized models pre-trained on similar or higher compute budgets, demonstrating that fully democratized, non-whitelisted participation is not only feasible, but can be achieved at unprecedented scale for a globally distributed pre-training run.
BibTeX
@misc{lidin2026covenant,
  author = {Joel Lidin and Amir Sarfi and Erfan Miahi and Quentin Anthony and Shivam Chauhan and Evangelos Pappas and Benjamin Therien and Eugene Belilovsky and Samuel Dare},
  title = {{Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet}},
  year = {2026},
  eprint = {2603.08163},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.08163},
}
Figure