TL;DR
The paper introduces ETTIN, the first open-source suite of perfectly paired encoder-only and decoder-only language models (ranging from 17M to 1B parameters) trained on up to 2 trillion tokens using identical data and hyperparameters. The study definitively shows that native architectures vastly outperform "cross-adapted" models—proving that a smaller, natively trained encoder easily beats a much larger decoder forced to do encoder tasks via continued training.
What Problem Does This Paper Address?
For years, the AI community has experienced an architectural schism. While decoder-only models (like GPT and Llama) dominate generative tasks, encoder-only models (like BERT) remain heavily used for classification and retrieval.
However, directly comparing these two paradigms has been nearly impossible. Previous comparisons were deeply confounded because the models being tested used completely different pre-training datasets, tokenizers, parameter counts, and optimization schedules. Furthermore, a growing sentiment suggests that large decoders can simply be adapted to encoder tasks through continued training. This paper introduces a controlled, "apples-to-apples" framework to test these assumptions rigorously.
Methods
The authors created the ETTIN suite, consisting of 10 models split into 5 identical pairs (17M, 32M, 68M, 150M, 400M, and 1B parameters).
To ensure a perfectly fair comparison, every single variable was held constant across each pair except for the attention pattern and the training objective:
- Encoders: Use bidirectional attention and Masked Language Modeling (MLM).
- Decoders: Use causal attention and Causal Language Modeling (CLM).
The training pipeline follows a modern three-phase recipe:
- Base Pre-training: 1.7 Trillion tokens with a trapezoidal learning rate schedule.
- Context Extension (Mid-Training): 250 Billion tokens, scaling context length up to 8,000 tokens using RoPE.
- Decay Phase: 50 Billion tokens using high-quality filtered data.
Additionally, the authors explored Cross-Objective Training by taking the finalized models and continuing their pre-training for another 50 Billion tokens on the reverse objective (e.g., training decoders with Masked Next Token Prediction to act like encoders).
Datasets
- Pre-training Data: A massive open-data mixture drawing from DCLM, Dolma v1.7, Starcoder, CC News, Open-Web-Math, Arxiv, Wikipedia, and Tulu Flan.
- Encoder Evaluation: GLUE (classification), MTEB v2 English (embeddings/retrieval), MDLR (long-context retrieval), and CodeSearchNet.
- Decoder Evaluation: Eleuther AI harness benchmarks including ARC, HellaSwag, LAMBADA, OpenBookQA, Social IQA, TriviaQA, Winogrande, and the Winograd Schema Challenge (WSC).
- Bias Evaluation: The WinoGender benchmark to analyze gender pronoun prediction biases.
Key Insights and Ideas
- One Recipe to Rule Them All: Remarkably, using the exact same training recipe yielded state-of-the-art (SOTA) results for both architectures at their respective sizes, outperforming specialized baselines like ModernBERT (encoder) and Llama 3.2 / SmolLM2 (decoders).
- The Inefficiency of Cross-Objective Adaptation: Trying to force a decoder to become an encoder (or vice versa) via continued training is highly inefficient. A 400M native encoder outperforms a 1B decoder that underwent 50B tokens of continued MLM training on classification tasks.
- Architectural Bias in Learning: Even when fed the exact same data, the underlying objective changes how models learn. On the WinoGender bias task, the MLM objective naturally biases encoders toward choosing gender-neutral pronouns, whereas decoders heavily favor male pronouns.
Key Results
The ETTIN suite consistently matched or established new SOTA performance marks for open-data models across both encoder and decoder tasks.
As illustrated in Figure 1, encoders dominate classification (MNLI) and retrieval (MS MARCO), while decoders excel at generative tasks. The gap widens as models scale, and cross-objective training (the dotted lines) fails to bridge the performance deficit to the native architecture.
- Encoder Wins: The Ettin-Enc-150M achieved a GLUE average score of 88.9, beating ModernBERT-base (88.4).
- Decoder Wins: Ettin-Dec-1B scored an average of 59.0 across generative tasks, outperforming Llama-3.2-1B (56.6).
Limitations and Future Work
- Compute Constraints at Scale: Due to hardware limitations, the largest 1B parameter models were only trained on 667 Billion tokens (1/3 of the full 2T dataset).
- Generative Encoder Evaluation: Evaluating encoder-only models on generative tasks requires iterative mask-filling techniques, which remain uncalibrated and could be improved in future iterations.
- Scale Limits: The suite currently tops out at 1B parameters. Testing whether these architectural gaps persist at the 3B or 7B parameter scale remains an open question for future research.
Summary updated