Introduction
In the current landscape of artificial intelligence, decoder-only architectures (the "GPT style") have become the de facto standard for most applications. While these models are celebrated for their generative capabilities, their performance on Natural Language Understanding (NLU) tasks—such as named entity recognition (NER) or linguistic acceptability—has not been rigorously compared to traditional encoder-based models (like BERT) across diverse languages. This paper addresses this gap by evaluating both architectures on an expanded version of the ScandEval benchmark, focusing on Germanic languages and challenging the assumption that larger parameter counts inherently lead to better language understanding.
Methodology
To facilitate a direct comparison between encoder models (which use classification heads) and decoder models (which generate text), the authors developed a "Generative NLU" framework.
- Task Re-framing: NLU tasks were converted into few-shot generative prompts. For instance, instead of predicting a class index, a decoder is prompted to output a specific label (e.g., "positive") or a structured JSON object for NER.
- Constrained Decoding: The researchers utilized the
outlinespackage to enforce structural constraints on decoder outputs, ensuring they followed parseable formats like JSON. This allowed for fair scoring against ground-truth classifications. - Mean Rank Score: To aggregate performance across heterogeneous tasks and languages, the study introduced a Mean Rank Score. This metric represents the average number of standard deviations a model's performance falls from the top-performing model in a given category, providing a normalized view of relative capability.
Experimental Setup
The study utilizes an expanded ScandEval benchmark covering eight languages: Danish, Swedish, Norwegian (Bokmål and Nynorsk), Icelandic, Faroese, German, Dutch, and English. The tasks span four core NLU domains: NER, Sentiment Classification, Linguistic Acceptability, and Extractive Question Answering (QA).
Key Results and Analysis
The Efficiency of Encoders
A primary finding is that smaller encoder models often outperform massive decoders in NLU tasks. In the English leaderboard, the state-of-the-art decoder GPT-4 was surpassed by both DeBERTa-v3-large and ELECTRA-base. This suggests that the bidirectional context of encoders provides a fundamental advantage for understanding-oriented tasks that autoregressive decoders struggle to match, even with significantly more parameters.
Task Specialization
The analysis revealed a clear architectural bias. Decoders emerged as "QA specialists," where the autoregressive pre-training objective aligns naturally with the question-answering format. Conversely, being a generative model was a strong predictor of lower performance in NER and Linguistic Acceptability.
Language-Specific Variations
Performance trends were not uniform across languages:
- Danish: Uniquely, decoders dominated the Danish leaderboard, led by GPT-4 and specialized Llama-3 variants.
- Icelandic: The competition was a stalemate, with mDeBERTa-v3-base and GPT-4-turbo achieving nearly identical top-tier scores.
- English: Encoders maintained a clear lead, though the authors note potential data leakage in sentiment datasets (like SST5) may have artificially boosted decoder scores.
Conclusion
This comparative analysis demonstrates that the "bigger is better" mantra of the decoder era does not always apply to NLU. Smaller, fine-tuned encoder models remain highly competitive and often superior for tasks requiring deep linguistic analysis rather than text generation. For practitioners, the choice between architectures should be driven by the specific task requirements—favoring decoders for QA-heavy applications and encoders for structured extraction and classification—while considering the significant efficiency gains offered by encoder-based models.
Summary updated