Zero-shot prompt-based classification: topic labeling in times of foundation models in German Tweets

Simon Münker, Kai Kugler, Achim RettingerACL 2025

TL;DR

This paper explores using large language models (LLMs) for zero-shot topic classification on German tweets, demonstrating that prompt-based approaches can achieve performance comparable to traditional fine-tuned models without requiring labeled training data. This work highlights a significant shift in NLP towards task unification and reducing reliance on extensive pre-labeled datasets, especially for real-world, multilingual scenarios.

What Problem Does This Paper Address?

Automating text annotation, especially for social media, is crucial for large-scale analysis. Traditionally, this involved supervised learning with labeled datasets, which are time-consuming and expensive to create. Recent advancements in LLMs offer a new paradigm: zero-shot or few-shot learning, where models can perform tasks based on natural language instructions (prompts) without task-specific training data. However, the effectiveness of these LLMs, particularly for non-English content and in noisy, real-world settings like German Twitter data concerning political crises, remains an open question. This paper aims to empirically evaluate prompt-based LLM classification against established methods, focusing on the impact of prompt detail and multilingual challenges.

Methods

The authors compare three main approaches:

  • Baseline: A Multinomial Naive Bayes classifier, representing the pre-deep learning era.
  • Fine-tuned Transformer: A BERT-based model ("gbert-base") fine-tuned on the specific task, including an additional domain adaptation phase using raw Twitter data. This represents the current state-of-the-art for traditional supervised learning.
  • Zero-Shot Prompting: Instruction-tuned LLMs (Flan-T5 and mT0) are used with varying levels of prompt detail. This approach leverages the models' general language understanding capabilities by providing task descriptions and class definitions directly in the prompt, bypassing the need for labeled training data. The authors experiment with different prompt formulations, ranging from a basic task name to a detailed "handbook" style description.

Datasets

The study uses a custom dataset of approximately 750,000 German tweets collected between October 2022 and May 2023, focusing on topics related to European energy crises. A subset of around 7,000 tweets was manually annotated by two domain experts. Notably, only about 3,000 of these tweets met the annotation criteria, highlighting the inherent noise and ambiguity in real-world social media data. The authors deliberately avoided academic benchmarks to simulate a more realistic scenario.

Key Insights and Ideas

  • Prompt Engineering Matters: The level of detail in prompts significantly impacts LLM performance, with larger models benefiting more from comprehensive instructions.
  • Multilingual Challenges: While LLMs can process German tweets and identify classes, their instruction-following capabilities for non-English tasks can be limited, especially for smaller models, suggesting that English-centric instruction tuning might not fully translate to other languages.
  • Zero-Shot Viability: Prompt-based LLMs can achieve performance comparable to fine-tuned models, even in challenging, non-English, real-world scenarios, signaling a paradigm shift in NLP.
  • Beyond Traditional Metrics: The free-form text output of LLMs necessitates a move away from standard confusion matrix-based metrics, requiring careful pre-processing and qualitative analysis.
  • Potential for Bias: LLMs inherit biases from their training data, which can be amplified and particularly problematic when applied to politically sensitive content, potentially imposing Anglo-centric interpretations on German discourse.

Key Results

  • The fine-tuned BERT model achieved an approximate weighted average F1 score of 86%, significantly outperforming the Naive Bayes baseline (66%).
  • The zero-shot prompt-based approaches, particularly with a detailed prompt (handbook), achieved performance close to the fine-tuned BERT model. For instance, Flan-T5 with the handbook prompt achieved a macro average F1 of 0.7686, while fine-tuned BERT was 0.8532.
  • Larger LLMs (XL versions) showed better understanding of German prompts compared to smaller models.
  • Simply mentioning the task name in the prompt (Prompt 2) already provided a noticeable performance boost for LLMs, suggesting they can leverage internal knowledge of similar tasks.

Limitations and Future Work

The authors acknowledge several limitations:

  • Language and Cultural Specificity: Findings might not generalize to other languages or cultures due to potential limitations in multilingual understanding and English-centric training.
  • Temporal Constraints: The data was collected during a specific period (European energy crisis), limiting temporal generalizability.
  • Annotation Subjectivity: The inherent subjectivity in topic classification, especially for political content, introduces variability.
  • Evaluation Methodology: The restriction to exact case-insensitive matches for LLM outputs might have been overly conservative.

Future work suggestions include:

  • Larger and More Capable LLMs: Exploring newer, more powerful LLMs to further improve zero-shot performance and enable more complex prompt variants.
  • Few-Shot Learning: Incorporating examples (few-shot) into prompts to enhance results and reduce inconsistencies and hallucinations.
  • Persona-Based Prompting: Adding personas to prompts or conditioning LLMs on individual human behavior to capture more nuanced annotation outcomes and address inconsistencies.
  • Bias Mitigation: Developing explicit bias mitigation strategies and considering the broader societal implications of automating political discourse analysis, including the potential for misuse.

Summary updated

BibTeX
@inproceedings{munker2025zero,
  author = {Simon Münker and Kai Kugler and Achim Rettinger},
  title = {{Zero-shot prompt-based classification: topic labeling in times of foundation models in German Tweets}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure