TL;DR
Large Language Models (LLMs) are notoriously "brittle," where even minor prompt changes can lead to wild performance swings. This paper explores this sensitivity in the context of Slovak—a low-resource language—finding that while native-language prompts offer modest gains, the most critical factor for success is the presence of explicit task instructions rather than few-shot examples.
What Problem Does This Paper Address?
While LLMs have revolutionized NLP, they remain highly sensitive to how a prompt is structured, formatted, and even which language is used for instructions. This "prompt sensitivity" is particularly acute for low-resource languages like Slovak, where models have less training data to fall back on.
The authors investigate how prompt design affects performance on specialized tasks related to migration discourse in Slovak media. They specifically look at three tasks:
- Theme Relevance: Is an article actually about human migration?
- Geographical Relevance: Does the text pertain specifically to Slovakia?
- Location Extraction: Identifying the source and destination of migration mentioned in a sentence.
The goal is to determine if prompting in the native language (Slovak) vs. English makes a difference, and which parts of a prompt (instructions vs. examples) are the real "load-bearing" components.
Methods
The researchers conducted a case study comparing "non-reasoning" models (like GPT-4o and Llama 3.3) against "reasoning" models (like Grok-3-mini and DeepSeek-r1).
Key methodological choices included:
- Language Testing: Comparing English "default" prompts against native Slovak prompts.
- Prompting Strategies: Evaluating Zero-Shot, Few-Shot, and RAG (Retrieval-Augmented Generation) approaches.
- Reasoning "Jailbreaks": For reasoning models, the authors had to use specific prompting techniques to "force" the models to perform their internal Chain-of-Thought (CoT) reasoning in Slovak rather than defaulting to English.
- Ablation Study: To find the source of "brittleness," they systematically removed the Task Description and/or the Examples from the prompts to see which removal caused the most damage.
Datasets
The study utilized a theme-specific Slovak annotated dataset focused on media portrayal of migration from 2003 to 2022:
- Theme Relevance Subset: Articles sampled from a massive 1.8-million-item corpus, annotated by at least three humans.
- Locality & Geo-Relevance Subset: A manually curated set of several thousand sentences from Slovak media (2022–2024) covering conflicts in Ukraine, Syria, and Gaza.
- Benchmarks: The authors compared results against existing benchmarks like SlovakBERT.
Key Insights and Ideas
- The "Instruction" Anchor: The most surprising finding was that explicit task instructions are far more important than few-shot examples. In many cases, adding examples actually hindered smaller models, whereas removing instructions caused performance to collapse.
- Native Language Edge: Slovak prompts generally outperformed English ones, particularly for structured tasks like location extraction. This suggests that for low-resource languages, the linguistic nuances captured in native instructions help the model navigate the task more accurately.
- Size Equals Robustness: Larger models (e.g., GPT-4o) are significantly more robust to prompt variations. While a "mini" model might see a 24% performance drop when instructions are removed, a full-sized model might only dip by 1%.
- Reasoning Volatility: Models forced to use Chain-of-Thought reasoning showed higher "volatility." They could reach higher performance peaks but also suffered from much lower "floors" (minimum performance) compared to standard models.
Key Results
The researchers used Macro F1 scores to evaluate performance across different prompt variants.
- Ablation Impact: As seen in Figure 1, removing both the task and examples ("none") led to a massive performance degradation of 50% to 77% across various GPT models.
- Extraction Gains: Location extraction saw a statistically significant improvement of +0.017 F1 points when using Slovak prompts over English ones.
- Model Comparison: Non-reasoning models generally provided more stable performance on these specialized tasks than reasoning-heavy models.
As shown in the table above (Figure 2), while English prompts are often competitive, the Slovak (SVK) versions frequently clinch the top performance (bolded values), especially in the "Loc ext." (Location Extraction) task.
Limitations and Future Work
The authors acknowledge several hurdles:
- Single-Run Constraints: Due to computational costs, experiments weren't replicated across multiple random seeds, which might mask some variance.
- Generalizability: Because the study is Slovak-specific, the results might not perfectly translate to other low-resource languages with different morphological structures.
- Annotation Agreement: The inter-annotator agreement for the theme relevance task was only "low-moderate" ($\alpha = 0.326$), suggesting that even for humans, these tasks are highly subjective.
Future work could involve "autoprompting" (letting the model optimize its own instructions) and exploring how these findings hold up across a broader array of low-resource linguistic contexts.
Summary updated