Form and Meaning in Intrinsic Multilingual Evaluations

Wessel Poelman, Miryam de Lhoneux

TL;DR

When evaluating multilingual language models, researchers often assume that using parallel datasets makes cross-lingual comparisons of metrics like perplexity fair because semantic meaning is held constant. This paper reveals that these intrinsic metrics measure information-theoretic content (form) rather than semantics, leading to severe ranking inconsistencies when sentences are paraphrased. Consequently, cross-lingual conclusions about which language is "harder" to model can completely flip based on the specific paraphrase chosen for the test set.

What Problem Does This Paper Address?

How do we know if a language model performs better in English than in German, or if one language is inherently harder to model than another? A standard practice in NLP is to compute intrinsic metrics like perplexity (PPL) or negative log-likelihood (NLL) on multi-parallel datasets—benchmarks where every sentence is translated across multiple languages. The underlying assumption is that because the semantic meaning is identical, any difference in the metric reflects true differences in language difficulty or model quality.

However, the authors point out a fundamental flaw: language models are optimized to minimize cross-entropy loss, which is an information-theoretic objective concerned with information content (the probability of tokens occurring), not semantic meaning. Parallel sentences ensure identical meaning, but they do not neutralize variations in form (vocabulary choices, syntax, and paraphrases). If an evaluation metric is highly sensitive to the specific form chosen, cross-lingual comparisons become unstable and potentially misleading.

Figure 1 – Each line represents a row in a parallel dataset, each dot is an individual sentence; four German and one English. Common multilingual evaluations cen- ter around comparing intrinsic metrics such as the neg- ative log-likelihood (NLL) or bits per character (BPC) with the assumption that these comparisons are fair since the semantic meaning is consistent. However, these metrics measure information content in the information- theoretic sense. This results in (1) differences within a language (DEi ↔DEk), and (2) inconsistency across languages: if the EN sentence falls outside the range of the DE sentences it is consistent. If it falls within the range it is inconsistent, meaning conclusions can flip depending on the DE sentence we choose in our test set.

Methods

The authors systematically evaluate six common intrinsic metrics: Negative Log-Likelihood (NLL), Perplexity (PPL), Bits per Character (BPC), Bits per English Character (BPEC), Information Parity (IP), and Mean Reciprocal Rank (MRR).

They train a series of tiny Llama-3-style causal language models (~143M parameters for monolingual models and ~279M for multilingual models) to observe metric behavior over training checkpoints.

To test the core issue of paraphrase consistency, they compare the metric scores of an English source sentence against four distinct, human-translated German paraphrases that convey the exact same meaning. They analyze consistency at two levels:

  1. Sample-level consistency: Checking if the relative ranking between the English source and the German translations remains stable across all four paraphrases for a given sentence.
  2. Split-level consistency: Checking if the overall dataset averages remain consistent when the paraphrases are grouped into different splits (e.g., sorting rows from "easiest" to "hardest" paraphrase).

Datasets

  • EuroParl: Used for training monolingual and multilingual models across 21 European languages (~211k parallel samples).
  • United Nations Parallel Corpus (UNPC): Used for training across 6 official UN languages (~11.2M parallel samples).
  • FLORES-200 (FLORES+): Used as a standard benchmark to track metrics across training checkpoints.
  • WMT-19 Paraphrases: A dataset containing high-quality news articles (~2000 samples) where each English source sentence has four distinct human-translated German references ($DE_1$ to $DE_4$). This served as the core dataset for testing paraphrase consistency.

Key Insights and Ideas

  • Form $\neq$ Meaning: Parallel data controls for semantic meaning, but language models evaluate the probability of the form. A paraphrase that means the exact same thing can have a vastly different probability under the model.
  • The Illusion of Dataset Fairness: Just because a benchmark like FLORES-200 is parallel doesn't mean it provides a neutral playing field. The specific translation choices made by the creators heavily bias the resulting perplexity scores.
  • Ranking Instability: In nearly half of the tested cases, a model might find German "easier" than English using one valid translation, but "harder" than English if a different, equally valid paraphrase is selected.
  • Averages Can Mask Flaws: While standard random splits might average out to look consistent, any systematic selection bias in how paraphrases are chosen can completely alter the evaluation outcome.

Key Results

  • High Sample-Level Inconsistency: Across both monolingual ($M^{\text{Mono}}$) and multilingual ($M^{\text{Multi}}$) models, approximately 47% to 52% of all sentence samples exhibited inconsistent rankings across paraphrases for NLL, BPC, and MRR.
  • Flipping Conclusions on Split-Level Averages: When evaluating the original splits from the WMT-19 dataset, the averages appeared consistent (English NLL was higher than all German splits). However, when the German paraphrases were sorted row-wise into "easiest" and "hardest" splits ($DE^S$), the English NLL (6.91) fell right in the middle of the German range (5.89 to 7.23). This means a researcher could conclude either that German is easier or harder to model than English, purely based on how the parallel test set was constructed.
Figure 3 – Paraphrase consistency of NLL values for the EN source and four parallel DE paraphrases using monolingual EuroParl models. On the x-axis, we list the sample index. Each sample consists of the EN sentence (green) and the four DE paraphrases (light red). For the sake of visual clarity, we sort the results by the English NLL. We show the ranking consistency: if all DE paraphrases are above or below the EN source, it means the ranking is consistent. Inconsistent rankings are marked with a red dot inside the green diamond for English. We see similar results for the multilingual model, as well as for the other metrics, see §C.2.

Limitations and Future Work

  • Model Scale: The experiments were conducted on tiny models (~143M–279M parameters). While small models are excellent for controlled scientific inquiries, future work needs to verify if massive large language models (LLMs) exhibit the same degree of sensitivity, though the underlying information-theoretic mismatch remains theoretically identical.
  • Data Constraints: The requirement for multi-parallel corpora for both training and evaluation limited the choice and size of the datasets used.
  • Clean Language Divisions: The paper treats languages as cleanly distinct entities, whereas real-world language distributions involve loan words, code-mixing, and shared token spaces, which introduces additional evaluation confounds.
  • Moving Forward: The authors suggest that intrinsic metrics might be fundamentally unsuited for answering broad cross-lingual questions like "which language is easier to model?". Instead, the field may need to pivot toward statistical linguistic laws or external databases of typologically diverse close translations that explicitly mark morphosyntactic and morphosemantic features.

Summary updated

Abstract

Intrinsic evaluation metrics for conditional language models, such as perplexity or bits-per-character, are widely used in both mono- and multilingual settings. These metrics are rather straightforward to use and compare in monolingual setups, but rest on a number of assumptions in multilingual setups. One such assumption is that comparing the perplexity of CLMs on parallel sentences is indicative of their quality since the information content (here understood as the semantic meaning) is the same. However, the metrics are inherently measuring information content in the information-theoretic sense. We make this and other such assumptions explicit and discuss their implications. We perform experiments with six metrics on two multi-parallel corpora both with mono- and multilingual models. Ultimately, we find that current metrics are not universally comparable. We look at the form-meaning debate to provide some explanation for this.
BibTeX
@inproceedings{poelman2026form,
  author = {Wessel Poelman and Miryam de Lhoneux},
  title = {{Form and Meaning in Intrinsic Multilingual Evaluations}},
  booktitle = {EACL (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.113},
}
Figure