Zuse: A Decision Model for European Languages

Umair Mehmood ImamarXiv 2026 · arXiv

Zuse: Bringing Decision Models to the European Linguistic Landscape

TL;DR

Zuse is a new family of "decision models" (4B to 27B parameters) designed specifically for 27 European languages. By shifting away from text generation to probability-based classification, Zuse provides a more reliable framework for operational tasks like moderation and routing, revealing that model reliability is driven more by task type than by language.

What Problem Does This Paper Address?

While generative Large Language Models (LLMs) dominate the headlines, many real-world applications don't actually need a model to write a poem or a summary. Tasks like ticket routing, policy checks, and form extraction require a choice among fixed options. "Decision models" are a specialized class of LLMs that read a document and return probabilities for specific options rather than generating free-form text.

Until now, these models (like Jev and Kev) have been evaluated almost exclusively on English. This leaves a massive gap for European organizations that operate across dozens of languages, some with very limited training data. The Zuse project aims to determine if the decision-model design actually transfers across the diverse linguistic landscape of Europe and what factors truly govern their reliability and automation potential.

Methods

The Zuse family consists of three models (4B, 9B, and 27B parameters) built on frozen, open-weight Qwen bases.

Instead of full fine-tuning, the researchers used a low-rank adapter (LoRA) and a pointer head. The adapter modifies how the base model interprets the request, while the pointer head replaces the standard text output with a probability distribution over the provided options.

Figure 1: Zuse architecture: document and question in, one probability per option out. The low-rank adapter sits inside the frozen base’s layers.

Key training features include:

  • Permutation-consistency loss: This penalizes the model if it changes its answer just because the options were reordered.
  • Injection augmentation: Training the model to ignore malicious instructions planted within a document.
  • Cached Encoding: In production, the document "state" is encoded once and cached, allowing multiple questions to be asked about the same text very efficiently.

Datasets

The models were evaluated on a massive scale:

  • European Suite: 11 sources covering 9 task groups and 27 languages (20,044 rows). This included benchmarks like Belebele (reading), SIB-200 (topics), and ScandEval (acceptability).
  • World Languages: A broader reach test across 210 languages using Belebele and SIB-200.
  • English Public Suites: Standard benchmarks like Transfer v4 and SemIf to compare against English-centric models like Jev and Kev.
Table 2: Evaluation suites and their use during development.

Key Insights and Ideas

  • Task > Language for Calibration: One of the most significant findings is that miscalibration (how much a model's confidence matches its actual accuracy) is structured by the task, not the language. If a model is overconfident on a knowledge quiz in English, it will be similarly overconfident in Estonian. This means developers can't use a single "temperature" setting to fix calibration across different tasks.
  • The "Base Model" Ceiling: The researchers found that a model's reach into obscure languages is largely determined by the underlying base model. Changing the training data moved scores by only ~2 points, while languages outside the base model's original training set trailed by 23 to 28 points.
  • Automation Limits: High accuracy doesn't always mean a task is ready for automation. The "automatable share" (how many cases can be handled without human intervention at a 5% error budget) correlates better with chance-corrected skill than raw accuracy. For example, Nordic grammar tasks had 72.6% accuracy but were only 8% automatable because the model's confidence didn't reliably separate right answers from wrong ones.
  • Label-Word Collision: A subtle pitfall in multi-source training is when different datasets use the same label words (e.g., "Topic") to mean different things. This "vocabulary conflict" can significantly degrade performance.

Key Results

  • Accuracy: The Zuse-27B model reached 87.5% accuracy on the European suite.
  • Language Performance: The model performed strongest on Polish and Bulgarian (>90%) and weakest on Nynorsk (~67%).
  • Automation: Zuse-27B showed high automation potential for RAG passage relevance (100%) and banking intents (99%), but struggled with legal rulings (2%).
Figure 2: Accuracy and in-sample automation share by task group, Zuse-27B.
  • Comparison: While Zuse leads in European language coverage, it still trails behind Jev and Kev on English-only public suites.

Limitations and Future Work

The authors acknowledge several hurdles:

  • Knowledge Weakness: The models remain relatively weak on "knowledge-heavy" questions (e.g., MMLU-Pro).
  • Selection Bias: Because the European suite was used during the development process to diagnose errors, the reported scores might be slightly optimistic.
  • Non-Disclosure: The specific composition of the training data remains commercially sensitive, which limits full reproducibility by the community.

Future work will focus on a "clean" evaluation using a completely new, unseen European test suite to resolve potential confounds identified in this study.

Summary updated

Abstract

Decision models read a document and return a probability for each of a fixed set of options instead of generating text. Several appeared in 2026, but none has published results outside English. We describe Zuse, three decision models (4B, 9B and 27B) that train a low-rank adapter and a pointer head on frozen open-weight bases, targeting 27 European languages, and use them to study how this design behaves across languages and tasks. On a broad European suite of 11 sources excluded from training, covering 9 task groups and 27 languages, a development set that also guided model selection, the 27B model reaches 87.5% accuracy with a pooled expected calibration error of 0.016, and 0.035 when weighted per source. Post-hoc analyses of these single-seed results give three findings. First, miscalibration is structured by task rather than language: on parallel reading and topic items in 26 languages the confidence gap varies about as much as sampling noise, while across sources it spans -0.07 to +0.09, with knowledge tasks overconfident and acceptability judgements underconfident at all three sizes, so one fitted temperature cannot correct both. Second, the share of decisions that can be automated at a 5% error budget follows chance-corrected skill (Spearman 0.97 over 33 source and model pairs) more closely than raw accuracy (0.91). Third, reach beyond the training languages is set by the base model: across 210 languages, changing the training run moved scores by at most about 2 points within a size, while languages outside the base's documented list trail by 23 to 28 points at every size. On English public suites Zuse is behind Jev and Kev, including Kev models on the same bases. We also report two pitfalls in building multi-source training data: label-word collision between topic schemes, and an injection augmentation that taught a shortcut. Zuse can be tried at https://bineric.com/zuse.
BibTeX
@misc{imam2026zuse,
  author = {Umair Mehmood Imam},
  title = {{Zuse: A Decision Model for European Languages}},
  year = {2026},
  eprint = {2611.06238},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06238},
}
Figure