Apertus: Democratizing Open and Compliant LLMs for Global Language Environments
Alejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Angelika Romanou, Antoni-Joan Solergibert, Barna Pásztor, Bettina Messmer, Dhia Garbaya, Eduard Frank Ďurech, Ido Hakimi, Juan Garcia Giraldo, Mete Ismayilzada, Negar Foroutan, Skander Moalla, Tiancheng Chen, Vinko Sabolčec, Yixuan Xu, Michael Aerni, Badr AlKhamissi, Inés Altemir Marinas, Mohammad Hossein Amani, Matin Ansaripour, Ilia Badanin, Harold Benoit, Emanuela Boros, Nicholas John Browning, Fabian Bösch, Maximilian Böther, Niklas Canova, Camille Challier, Clément Charmillot, Jonathan Coles, Jan Milan Deriu, Arnout Devos, Lukas Drescher, Daniil Dzenhaliou, Maud Ehrmann, Dongyang Fan, Simin Fan, Silin Gao, Miguel Gila, María Grandury, Diba Hashemi, Alexander Miserlis Hoyle, Jiaming Jiang, Mark Klein, Andrei Kucharavy, Anastasiia Kucherenko, Frederike Lübeck, Roman Machacek, Theofilos Ioannis Manitaras, Andreas Marfurt, Kyle Matoba, Simon Matrenok, Henrique Mendonça, Fawzi Roberto Mohamed, Syrielle Montariol, Luca Mouchel, Sven Najem-Meyer, Jingwei Ni, Gennaro Oliva, Matteo Pagliardini, Elia Palme, Andrei Panferov, Léo Paoletti, Marco Passerini, Ivan Pavlov, Auguste Poiroux, Kaustubh Ponkshe, Nathan Ranchin, Javier Rando, Mathieu Sauser, Jakhongir Saydaliev, Mukhammadali Sayfiddinov, Marian Schneider, Stefano Schuppli, Marco Scialanga, Andrei Semenov, Kumar Shridhar, Raghav Singhal, Anna Sotnikova, Alexander Sternfeld, Ayush Kumar Tarun, Paul Teiletche, Jannis Vamvas, Xiaozhe Yao, Hao Zhao, Alexander Ilic, Ana Klimovic, Andreas Krause, Caglar Gulcehre, David Rosenthal, Elliott Ash, Florian Tramèr, Joost VandeVondele, Livio Veraldi, Martin Rajman, Thomas C. Schulthess, Torsten Hoefler, Antoine Bosselut
In the rapidly evolving landscape of Large Language Models (LLMs), the term "open" has become increasingly ambiguous. Many popular models are merely "open-weight," meaning their internal parameters are public, but the data and recipes used to create them remain locked behind corporate doors. Apertus changes that.
TL;DR
Apertus is a suite of fully open 8B and 70B parameter LLMs that prioritize data responsibility and massive multilinguality. Trained on 15T tokens across 1,800+ languages, Apertus sets a new benchmark for transparency by retroactively respecting web-crawling opt-outs and implementing novel techniques to prevent data memorization.
What Problem Does This Paper Address?
The authors identify a "responsibility gap" in the current open-model ecosystem. While models like Llama or Qwen provide weights, they often lack reproducible data pipelines and may include non-permissive or illegal material. Furthermore, most models are heavily biased toward English or a small subset of high-resource languages.
Apertus tackles three main challenges:
Data Compliance: Respecting the rights of content owners (via robots.txt) even after data has been crawled.
Verbatim Memorization: Preventing models from "regurgitating" sensitive or copyrighted training data.
Global Representation: Expanding high-quality LLM capabilities to lower-resource language environments.
Methods
Apertus is a dense decoder-only Transformer, but it introduces several architectural and procedural innovations to improve stability and ethics:
xIELU Activation: A new activation function that improves training efficiency by 30–40% compared to standard baselines.
Goldfish Loss: To prevent memorization, the authors adopt a training objective that randomly masks a small percentage (2%) of tokens. This prevents the model from learning exact token-to-context mappings, effectively suppressing verbatim recall while maintaining performance.
AdEMAMix Optimizer: This is the first time this optimizer—which uses a long-term exponential moving average of gradients—has been used for an LLM at this scale.
QRPO (Quantile Reward Policy Optimization): For preference alignment, they use QRPO, which optimizes an absolute reward signal rather than just relative preferences, making it more stable than traditional methods like DPO.
Datasets
The models were pretrained on a massive 15-trillion token corpus.
Sources: Primarily FineWeb and FineWeb-2, supplemented with StarCoder (code), FineMath, and DCLM-Edu.
Multilinguality: The corpus covers 1,811 languages, with ~40% of the data dedicated to non-English content.
New Benchmark: The authors introduce SwitzerlandQA, a novel benchmark specifically designed to test regional knowledge (geography, history, and civic life) across five languages, including Romansh.
Key Insights and Ideas
Retroactive Consent: The team retrieved robots.txt files as of January 2025 and retroactively removed blocked content from web scrapes dating back to 2013. This "hindsight" approach resulted in an 8% token loss in English data but ensures the model respects current owner preferences.
The "Compliance Tax": The authors honestly report that strict license filtering in post-training reduced average performance by about 7%. This highlights a significant tension between building the most capable model and building the most ethical one.
Memorization is Optional: As shown in Figure 2, the Goldfish objective successfully keeps verbatim memorization at baseline levels, even for sequences the model saw 128 times during training.
Apertus remains at baseline memorization levels regardless of how many times it saw a sequence during training.
Key Results
Apertus-70B is the first fully open model of its scale. Quantitatively, it rivals or surpasses open-weight counterparts on multilingual benchmarks.
Figure 1: Pretraining Evaluation Throughout Training. Comparison of downstream evaluation results across
model checkpoints as training progresses. Global Evaluation uses the full suite of evaluation benchmarks. English
includes only English-language tasks. For Global, the aggregation between different benchmarks consists of a
macro aggregation, where each different language of each dataset is considered as a separate datapoint to aggregate.
As seen in the group averages (Figure 4), Apertus-8B-Instruct is particularly strong in knowledge recall and cultural knowledge, outperforming other fully open models like OLMo-2 and EuroLLM.
Apertus models lead the "Fully Open" category in several key metrics, particularly cultural knowledge and instruction following.
Limitations and Future Work
While Apertus excels in multilinguality and responsibility, it lags slightly behind the very best open-weight models in mathematics and coding. The authors attribute this to a lack of specialized Reinforcement Learning with verifiers (RLVR), which they plan to implement in future versions.
Other future directions include:
Scaling: Moving beyond the 70B scale.
Distillation: Creating smaller, more efficient versions of the 70B model for edge devices.
Multimodality: Extending the Apertus compliance standards to vision and audio data.
By releasing not just the weights, but the full data pipeline and training code, the Apertus team has provided a rare, truly transparent foundation for the global AI community.
Summary updated
BibTeX
@inproceedings{hernandezcano2026apertus,
author = {Alejandro Hernández-Cano and Alexander Hägele and Allen Hao Huang and Angelika Romanou and Antoni-Joan Solergibert and Barna Pásztor and Bettina Messmer and Dhia Garbaya and Eduard Frank Ďurech and Ido Hakimi and Juan Garcia Giraldo and Mete Ismayilzada and Negar Foroutan and Skander Moalla and Tiancheng Chen and Vinko Sabolčec and Yixuan Xu and Michael Aerni and Badr AlKhamissi and Inés Altemir Marinas and Mohammad Hossein Amani and Matin Ansaripour and Ilia Badanin and Harold Benoit and Emanuela Boros and Nicholas John Browning and Fabian Bösch and Maximilian Böther and Niklas Canova and Camille Challier and Clément Charmillot and Jonathan Coles and Jan Milan Deriu and Arnout Devos and Lukas Drescher and Daniil Dzenhaliou and Maud Ehrmann and Dongyang Fan and Simin Fan and Silin Gao and Miguel Gila and María Grandury and Diba Hashemi and Alexander Miserlis Hoyle and Jiaming Jiang and Mark Klein and Andrei Kucharavy and Anastasiia Kucherenko and Frederike Lübeck and Roman Machacek and Theofilos Ioannis Manitaras and Andreas Marfurt and Kyle Matoba and Simon Matrenok and Henrique Mendonça and Fawzi Roberto Mohamed and Syrielle Montariol and Luca Mouchel and Sven Najem-Meyer and Jingwei Ni and Gennaro Oliva and Matteo Pagliardini and Elia Palme and Andrei Panferov and Léo Paoletti and Marco Passerini and Ivan Pavlov and Auguste Poiroux and Kaustubh Ponkshe and Nathan Ranchin and Javier Rando and Mathieu Sauser and Jakhongir Saydaliev and Mukhammadali Sayfiddinov and Marian Schneider and Stefano Schuppli and Marco Scialanga and Andrei Semenov and Kumar Shridhar and Raghav Singhal and Anna Sotnikova and Alexander Sternfeld and Ayush Kumar Tarun and Paul Teiletche and Jannis Vamvas and Xiaozhe Yao and Hao Zhao and Alexander Ilic and Ana Klimovic and Andreas Krause and Caglar Gulcehre and David Rosenthal and Elliott Ash and Florian Tramèr and Joost VandeVondele and Livio Veraldi and Martin Rajman and Thomas C. Schulthess and Torsten Hoefler and Antoine Bosselut},
title = {{Apertus: Democratizing Open and Compliant LLMs for Global Language Environments}},
booktitle = {Annual Meeting of the Association for Computational Linguistics},
year = {2026},
doi = {10.18653/v1/2026.acl-long.2172},
}