Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks

Dan Saattrup Nielsen, Kenneth C. Enevoldsen, Peter Schneider-KamparXiv 2024 · DOI

Introduction

In the current landscape of artificial intelligence, decoder-only architectures (the "GPT style") have become the de facto standard for most applications. While these models are celebrated for their generative capabilities, their performance on Natural Language Understanding (NLU) tasks—such as named entity recognition (NER) or linguistic acceptability—has not been rigorously compared to traditional encoder-based models (like BERT) across diverse languages. This paper addresses this gap by evaluating both architectures on an expanded version of the ScandEval benchmark, focusing on Germanic languages and challenging the assumption that larger parameter counts inherently lead to better language understanding.

Methodology

To facilitate a direct comparison between encoder models (which use classification heads) and decoder models (which generate text), the authors developed a "Generative NLU" framework.

  1. Task Re-framing: NLU tasks were converted into few-shot generative prompts. For instance, instead of predicting a class index, a decoder is prompted to output a specific label (e.g., "positive") or a structured JSON object for NER.
  2. Constrained Decoding: The researchers utilized the outlines package to enforce structural constraints on decoder outputs, ensuring they followed parseable formats like JSON. This allowed for fair scoring against ground-truth classifications.
  3. Mean Rank Score: To aggregate performance across heterogeneous tasks and languages, the study introduced a Mean Rank Score. This metric represents the average number of standard deviations a model's performance falls from the top-performing model in a given category, providing a normalized view of relative capability.

Experimental Setup

The study utilizes an expanded ScandEval benchmark covering eight languages: Danish, Swedish, Norwegian (Bokmål and Nynorsk), Icelandic, Faroese, German, Dutch, and English. The tasks span four core NLU domains: NER, Sentiment Classification, Linguistic Acceptability, and Extractive Question Answering (QA).

Overview of the datasets and task configurations used in the benchmark, including language distribution and shot counts for few-shot evaluation.

Key Results and Analysis

The Efficiency of Encoders

A primary finding is that smaller encoder models often outperform massive decoders in NLU tasks. In the English leaderboard, the state-of-the-art decoder GPT-4 was surpassed by both DeBERTa-v3-large and ELECTRA-base. This suggests that the bidirectional context of encoders provides a fundamental advantage for understanding-oriented tasks that autoregressive decoders struggle to match, even with significantly more parameters.

English leaderboard showing encoder models (marked with a red X in the Decoder column) outperforming top-tier decoder models (marked with a green check).

Task Specialization

The analysis revealed a clear architectural bias. Decoders emerged as "QA specialists," where the autoregressive pre-training objective aligns naturally with the question-answering format. Conversely, being a generative model was a strong predictor of lower performance in NER and Linguistic Acceptability.

Language-Specific Variations

Performance trends were not uniform across languages:

  • Danish: Uniquely, decoders dominated the Danish leaderboard, led by GPT-4 and specialized Llama-3 variants.
  • Icelandic: The competition was a stalemate, with mDeBERTa-v3-base and GPT-4-turbo achieving nearly identical top-tier scores.
  • English: Encoders maintained a clear lead, though the authors note potential data leakage in sentiment datasets (like SST5) may have artificially boosted decoder scores.

Conclusion

This comparative analysis demonstrates that the "bigger is better" mantra of the decoder era does not always apply to NLU. Smaller, fine-tuned encoder models remain highly competitive and often superior for tasks requiring deep linguistic analysis rather than text generation. For practitioners, the choice between architectures should be driven by the specific task requirements—favoring decoders for QA-heavy applications and encoders for structured extraction and classification—while considering the significant efficiency gains offered by encoder-based models.

Summary updated

Abstract

This paper explores the performance of encoder and decoder language models on multilingual Natural Language Understanding (NLU) tasks, with a broad focus on Germanic languages. Building upon the ScandEval benchmark, initially restricted to evaluating encoder models, we extend the evaluation framework to include decoder models. We introduce a method for evaluating decoder models on NLU tasks and apply it to the languages Danish, Swedish, Norwegian, Icelandic, Faroese, German, Dutch, and English. Through a series of experiments and analyses, we also address research questions regarding the comparative performance of encoder and decoder models, the impact of NLU task types, and the variation across language resources. Our findings reveal that encoder models can achieve significantly better NLU performance than decoder models despite having orders of magnitude fewer parameters. Additionally, we investigate the correlation between decoders and task performance via a UMAP analysis, shedding light on the unique capabilities of decoder and encoder models. This study contributes to a deeper understanding of language model paradigms in NLU tasks and provides valuable insights for model selection and evaluation in multilingual settings.
BibTeX
@misc{nielsen2024encoder,
  author = {Dan Saattrup Nielsen and Kenneth C. Enevoldsen and Peter Schneider-Kamp},
  title = {{Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks}},
  year = {2024},
  eprint = {2406.13469},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.13469},
}
Figure