The Brittle Compass: Navigating LLM Prompt Sensitivity in Slovak Migration Media Discourse

Samuel Harvan, Jaroslav Kopčan, Marek Šuppa, Andrej FindorarXiv 2026 · DOI · arXiv

TL;DR

Large Language Models (LLMs) are notoriously "brittle," where even minor prompt changes can lead to wild performance swings. This paper explores this sensitivity in the context of Slovak—a low-resource language—finding that while native-language prompts offer modest gains, the most critical factor for success is the presence of explicit task instructions rather than few-shot examples.

What Problem Does This Paper Address?

While LLMs have revolutionized NLP, they remain highly sensitive to how a prompt is structured, formatted, and even which language is used for instructions. This "prompt sensitivity" is particularly acute for low-resource languages like Slovak, where models have less training data to fall back on.

The authors investigate how prompt design affects performance on specialized tasks related to migration discourse in Slovak media. They specifically look at three tasks:

  1. Theme Relevance: Is an article actually about human migration?
  2. Geographical Relevance: Does the text pertain specifically to Slovakia?
  3. Location Extraction: Identifying the source and destination of migration mentioned in a sentence.

The goal is to determine if prompting in the native language (Slovak) vs. English makes a difference, and which parts of a prompt (instructions vs. examples) are the real "load-bearing" components.

Methods

The researchers conducted a case study comparing "non-reasoning" models (like GPT-4o and Llama 3.3) against "reasoning" models (like Grok-3-mini and DeepSeek-r1).

Key methodological choices included:

  • Language Testing: Comparing English "default" prompts against native Slovak prompts.
  • Prompting Strategies: Evaluating Zero-Shot, Few-Shot, and RAG (Retrieval-Augmented Generation) approaches.
  • Reasoning "Jailbreaks": For reasoning models, the authors had to use specific prompting techniques to "force" the models to perform their internal Chain-of-Thought (CoT) reasoning in Slovak rather than defaulting to English.
  • Ablation Study: To find the source of "brittleness," they systematically removed the Task Description and/or the Examples from the prompts to see which removal caused the most damage.

Datasets

The study utilized a theme-specific Slovak annotated dataset focused on media portrayal of migration from 2003 to 2022:

  • Theme Relevance Subset: Articles sampled from a massive 1.8-million-item corpus, annotated by at least three humans.
  • Locality & Geo-Relevance Subset: A manually curated set of several thousand sentences from Slovak media (2022–2024) covering conflicts in Ukraine, Syria, and Gaza.
  • Benchmarks: The authors compared results against existing benchmarks like SlovakBERT.

Key Insights and Ideas

  • The "Instruction" Anchor: The most surprising finding was that explicit task instructions are far more important than few-shot examples. In many cases, adding examples actually hindered smaller models, whereas removing instructions caused performance to collapse.
  • Native Language Edge: Slovak prompts generally outperformed English ones, particularly for structured tasks like location extraction. This suggests that for low-resource languages, the linguistic nuances captured in native instructions help the model navigate the task more accurately.
  • Size Equals Robustness: Larger models (e.g., GPT-4o) are significantly more robust to prompt variations. While a "mini" model might see a 24% performance drop when instructions are removed, a full-sized model might only dip by 1%.
  • Reasoning Volatility: Models forced to use Chain-of-Thought reasoning showed higher "volatility." They could reach higher performance peaks but also suffered from much lower "floors" (minimum performance) compared to standard models.

Key Results

The researchers used Macro F1 scores to evaluate performance across different prompt variants.

Figure 1: Distribution of macro F1 across prompt vari- ants. For each model panel, we plot max min macro F1 (bars) per model and overlay per variant absolute macro F1 (points) on a twin top x axis, with thin lines showing the min–max span. Models are alphabetized.
  • Ablation Impact: As seen in Figure 1, removing both the task and examples ("none") led to a massive performance degradation of 50% to 77% across various GPT models.
  • Extraction Gains: Location extraction saw a statistically significant improvement of +0.017 F1 points when using Slovak prompts over English ones.
  • Model Comparison: Non-reasoning models generally provided more stable performance on these specialized tasks than reasoning-heavy models.
Table 1: Results comparing different LLMs across tasks.

As shown in the table above (Figure 2), while English prompts are often competitive, the Slovak (SVK) versions frequently clinch the top performance (bolded values), especially in the "Loc ext." (Location Extraction) task.

Limitations and Future Work

The authors acknowledge several hurdles:

  • Single-Run Constraints: Due to computational costs, experiments weren't replicated across multiple random seeds, which might mask some variance.
  • Generalizability: Because the study is Slovak-specific, the results might not perfectly translate to other low-resource languages with different morphological structures.
  • Annotation Agreement: The inter-annotator agreement for the theme relevance task was only "low-moderate" ($\alpha = 0.326$), suggesting that even for humans, these tasks are highly subjective.

Future work could involve "autoprompting" (letting the model optimize its own instructions) and exploring how these findings hold up across a broader array of low-resource linguistic contexts.

Summary updated

Abstract

In this work, we present a case study that explores various tasks centered around the topic of migration in Slovak, a low-resource language, such as topic relevance and geographical relevance classification, and migration source/destination location term extraction. Our results demonstrate that native (Slovak) prompts yield a modest, task-dependent gains, while large models show significant robustness to prompt variations compared to their smaller counterparts. Analysis reveals that instructions (system or task) emerge as the most critical prompt component, more so than the examples sections, with task-specific performance benefits being more pronounced than overall language effects.
BibTeX
@misc{harvan2026brittle,
  author = {Samuel Harvan and Jaroslav Kopčan and Marek Šuppa and Andrej Findor},
  title = {{The Brittle Compass: Navigating LLM Prompt Sensitivity in Slovak Migration Media Discourse}},
  year = {2026},
  eprint = {2611.01219},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01219},
}

Loading related content...

Figure