Abstract
The PLLuM project introduces PLLuMIC, a high-quality instruction dataset designed to bridge the gap between generic, English-centric AI and linguistically nuanced Polish models. By combining human-authored "organic" prompts with converted NLP datasets and synthetic distillation, the researchers demonstrate that true linguistic adaptation requires more than just translation—it requires cultural and stylistic "re-tuning." Our results show significant improvements in linguistic competency and safety across model scales.
Introduction
Most state-of-the-art Large Language Models (LLMs) are "English-first." While they can speak Polish, they often do so with a "translation accent"—using unidiomatic phrasing, incorrect e-mail etiquette, or English-style punctuation. Furthermore, the "secret sauce" of instruction tuning is rarely shared, making it difficult for the research community to replicate model capabilities in non-English languages. The PLLuM team addresses this lack of transparency and the "linguistic interference" from English by developing a specialized Polish instruction corpus.
Methodology and Dataset Construction
The researchers developed a functional typology to categorize instructions based on their origin:
- Organic: Hand-crafted by over 50 professional linguists and annotators to capture natural human-model interactions.
- Converted: Automatically transformed from 33 existing high-quality Polish NLP resources (e.g., DIABIZ, NKJP-NER, PolQA) into a question-answer format.
- Synthetic: Generated by "strong" models (like Mixtral or Llama 3.1 70B) through multi-step pipelines, including RAG based on Polish government documents.
The team followed a two-stage adaptation process: Continued Pre-training (150 billion tokens of raw Polish text) followed by Instruction Tuning and Alignment using the ORPO (Odds Ratio Preference Optimization) algorithm. This ensures the models are not only linguistically competent but also safe and helpful.
Key Insights and Results
A critical finding was the "English Register" trap: human annotators often mimicked the style of English-to-Polish translations. Training was required to avoid this "AI-speak" and un-learn formulaic habits, such as stiff email greetings. The researchers also found that instruction tuning only works effectively if the model has been "primed" with massive amounts of Polish text during continued pre-training; without this foundation, instruction tuning can actually degrade performance.
Benchmark Performance
The adapted models (ranging from 8B to 70B parameters) significantly outperformed their base versions on the PLCC (Polish Linguistic and Cultural Competency) benchmark. For instance, Llama-PLLuM-70B-chat scored 66.33, compared to 47.83 for the original Llama-3.1-70B-Instruct.
Beyond linguistic competency, the team achieved remarkable results in safety. Through rigorous red-teaming, the Attack Success Rate (ASR)—the frequency with which a model follows a harmful prompt—was reduced from 22.27% in the base 70B model to just 0.79% in the aligned PLLuM version. On academic benchmarks (LLMzSzŁ), the 70B PLLuM model scored 64.42, nearly matching much larger proprietary models.
Limitations and Future Work
While the released subset of PLLuMIC is a valuable "gold standard," it represents only a fraction of the full 77k-instruction training set. The authors also caution against "recursive distillation"—training models on data generated by other models—which can lead to model collapse. Future efforts will focus on releasing a "synthetic extension" (PLLuMIC-syn-ext) to complement the currently available organic subset.
Availability: The manually annotated sample of PLLuMIC is available on Hugging Face at pelcra/PLLuMIC.