Zuse: Bringing Decision Models to the European Linguistic Landscape
TL;DR
Zuse is a new family of "decision models" (4B to 27B parameters) designed specifically for 27 European languages. By shifting away from text generation to probability-based classification, Zuse provides a more reliable framework for operational tasks like moderation and routing, revealing that model reliability is driven more by task type than by language.
What Problem Does This Paper Address?
While generative Large Language Models (LLMs) dominate the headlines, many real-world applications don't actually need a model to write a poem or a summary. Tasks like ticket routing, policy checks, and form extraction require a choice among fixed options. "Decision models" are a specialized class of LLMs that read a document and return probabilities for specific options rather than generating free-form text.
Until now, these models (like Jev and Kev) have been evaluated almost exclusively on English. This leaves a massive gap for European organizations that operate across dozens of languages, some with very limited training data. The Zuse project aims to determine if the decision-model design actually transfers across the diverse linguistic landscape of Europe and what factors truly govern their reliability and automation potential.
Methods
The Zuse family consists of three models (4B, 9B, and 27B parameters) built on frozen, open-weight Qwen bases.
Instead of full fine-tuning, the researchers used a low-rank adapter (LoRA) and a pointer head. The adapter modifies how the base model interprets the request, while the pointer head replaces the standard text output with a probability distribution over the provided options.
Key training features include:
- Permutation-consistency loss: This penalizes the model if it changes its answer just because the options were reordered.
- Injection augmentation: Training the model to ignore malicious instructions planted within a document.
- Cached Encoding: In production, the document "state" is encoded once and cached, allowing multiple questions to be asked about the same text very efficiently.
Datasets
The models were evaluated on a massive scale:
- European Suite: 11 sources covering 9 task groups and 27 languages (20,044 rows). This included benchmarks like Belebele (reading), SIB-200 (topics), and ScandEval (acceptability).
- World Languages: A broader reach test across 210 languages using Belebele and SIB-200.
- English Public Suites: Standard benchmarks like Transfer v4 and SemIf to compare against English-centric models like Jev and Kev.
Key Insights and Ideas
- Task > Language for Calibration: One of the most significant findings is that miscalibration (how much a model's confidence matches its actual accuracy) is structured by the task, not the language. If a model is overconfident on a knowledge quiz in English, it will be similarly overconfident in Estonian. This means developers can't use a single "temperature" setting to fix calibration across different tasks.
- The "Base Model" Ceiling: The researchers found that a model's reach into obscure languages is largely determined by the underlying base model. Changing the training data moved scores by only ~2 points, while languages outside the base model's original training set trailed by 23 to 28 points.
- Automation Limits: High accuracy doesn't always mean a task is ready for automation. The "automatable share" (how many cases can be handled without human intervention at a 5% error budget) correlates better with chance-corrected skill than raw accuracy. For example, Nordic grammar tasks had 72.6% accuracy but were only 8% automatable because the model's confidence didn't reliably separate right answers from wrong ones.
- Label-Word Collision: A subtle pitfall in multi-source training is when different datasets use the same label words (e.g., "Topic") to mean different things. This "vocabulary conflict" can significantly degrade performance.
Key Results
- Accuracy: The Zuse-27B model reached 87.5% accuracy on the European suite.
- Language Performance: The model performed strongest on Polish and Bulgarian (>90%) and weakest on Nynorsk (~67%).
- Automation: Zuse-27B showed high automation potential for RAG passage relevance (100%) and banking intents (99%), but struggled with legal rulings (2%).
- Comparison: While Zuse leads in European language coverage, it still trails behind Jev and Kev on English-only public suites.
Limitations and Future Work
The authors acknowledge several hurdles:
- Knowledge Weakness: The models remain relatively weak on "knowledge-heavy" questions (e.g., MMLU-Pro).
- Selection Bias: Because the European suite was used during the development process to diagnose errors, the reported scores might be slightly optimistic.
- Non-Disclosure: The specific composition of the training data remains commercially sensitive, which limits full reproducibility by the community.
Future work will focus on a "clean" evaluation using a completely new, unseen European test suite to resolve potential confounds identified in this study.
Summary updated