Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applications

Fadel M. Megahed, Ying-Ju Chen, L. Allision Jones-Farmer, Younghwa Lee, Jiawei Brooke Wang, Inez Maria ZwetslootarXiv 2025 · DOI · arXiv

TL;DR

This paper introduces a robust framework for evaluating the reliability and validity of Large Language Models (LLMs) in binary text classification tasks. It addresses the critical need for standardized methods to ensure LLM outputs are consistent and trustworthy, particularly in decision-support applications. The framework, tested on financial news sentiment analysis, reveals that smaller LLMs can perform comparably to larger ones in terms of reliability, challenging the assumption that bigger is always better.

What Problem Does This Paper Address?

The rapid adoption of LLMs for tasks like text classification has outpaced the development of standardized methods for evaluating their reliability and validity. Researchers and practitioners face a "wild west" scenario where the quality and trustworthiness of LLM-generated annotations are often uncertain. Existing methods for assessing reliability, primarily developed for human annotators, don't always translate well to the unique characteristics of LLMs, such as their stochastic nature and potential for generating invalid responses. This lack of established practices makes it difficult to select appropriate LLMs, determine adequate sample sizes for evaluation, and ensure that the LLM's outputs are consistent and dependable for critical decision-making.

Methods

The authors propose a four-phase framework for evaluating LLM consistency:

  1. Planning Phase: This involves determining the minimum sample size required for reliable estimates, selecting candidate LLMs, and designing effective prompts. They adapt psychometric principles to calculate sample sizes based on desired margins of error and confidence levels, considering metrics like Percent Agreement, Gwet's AC1, and Brennan-Prediger Coefficient.
  2. Data Collection Phase: This phase focuses on curating a representative dataset, setting up the LLM classification system (including data management, replication logic, and LLM interaction layers), and running multiple replicates for each LLM.
  3. Reliability Analysis Phase: This is where the core evaluation happens. The framework assesses both intra-rater reliability (consistency of a single LLM across multiple runs) and inter-rater reliability (agreement between different LLMs). They employ five complementary metrics: Conger's Kappa, Fleiss' Kappa, Gwet's AC1, Brennan-Prediger Coefficient, and Krippendorff's Alpha. Crucially, they propose methods to handle "invalid" or non-compliant LLM responses, using both NA-dropped (optimistic) and NA-penalized (conservative) approaches.
  4. Validity Analysis Phase: This phase assesses how well the LLM-generated classifications align with external benchmarks or ground truth. In their case study, they compare LLM outputs against both API-provided sentiment labels and actual stock market movements.

Datasets

  • StockNewsAPI: This dataset was used to extract financial news articles for sentiment classification. It provided article URLs, headlines, full text, publication source, date, related topics, API-provided sentiment classification, content type, and associated stock ticker symbols. The initial dataset contained 10,000 articles, later downsampled to 1,350 articles (equally divided between positive and negative sentiment) for the LLM evaluation. The API-provided sentiment labels served as the "benchmark" for evaluation.
  • Actual Market Behavior: This external criterion was used to assess predictive validity. It involved comparing LLM sentiment predictions against actual stock price movements (excess return relative to the S&P500 index) one trading day after the article's publication.

Key Insights and Ideas

  • Standardization is Crucial: The paper highlights the urgent need for standardized, psychometrically-grounded methods to evaluate LLMs in classification tasks, moving beyond ad-hoc approaches.
  • Smaller LLMs Can Be Mighty: Contrary to the "bigger is better" assumption, smaller and less expensive LLMs (like Claude-3-5-haiku and Gemma 3:1B) demonstrated reliability and accuracy comparable to their larger counterparts, challenging the FOMO mentality in LLM adoption.
  • Consistency Doesn't Guarantee Predictive Power: LLMs can be highly consistent in their classifications (high intra-rater reliability), but this doesn't automatically translate to accurate predictions of real-world outcomes like stock price movements. This suggests a limitation in LLMs' ability to perform complex causal inference.
  • Handling Invalid Responses is Key: LLMs often produce "invalid" or unexpected outputs. The framework's approach to systematically handling these (NA-penalized and NA-dropped analyses) is vital for accurate reliability assessment.
  • Inter-LLM Agreement is High, but Decreases with More Models: While individual LLMs are highly consistent, adding more LLMs to an ensemble generally leads to a slight decrease in inter-rater reliability, a predictable outcome.

Key Results

  • High Intra-Rater Reliability: Most LLMs achieved perfect agreement on 88-98% of articles across five replicates, with mean agreement rates exceeding 0.93.
  • Strong Inter-Rater Reliability: Even with combinations of multiple LLMs, reliability metrics generally remained high, often exceeding 0.70.
  • Benchmark Performance: LLMs demonstrated strong performance against the StockNewsAPI benchmark, with mean accuracy values ranging from 0.76 to 0.88. Smaller models like Gemma 3:1B, Llama 3.2:1B, and Claude-3-5-haiku often outperformed their larger counterparts.
  • Market Prediction Failure: LLMs performed at chance level (around 0.50) when predicting actual market movements, indicating that news sentiment alone, as captured by LLMs, is not a strong predictor of stock price changes. This aligns with the efficient market hypothesis.
  • Invalid Labels: Across all models, approximately 2.37% of labels were classified as "invalid" (e.g., "neutral," "unsure," or unparseable), with cheaper models showing slightly higher rates.

Limitations and Future Work

  • Binary Classification Focus: The current framework is designed for binary classification. Future work could extend it to multiclass settings, which introduce greater complexity for reliability metrics and sample size calculations.
  • Fuzzy/Graded Annotations: The framework could be adapted for settings where LLMs provide confidence scores or degrees of membership rather than discrete labels.
  • Domain Specificity: The case study focused on financial sentiment. Further research is needed to assess how reliability patterns vary across different domains (e.g., legal, medical) and how model size influences performance in these contexts.
  • Handling Non-Compliant Responses: While the paper addresses this, more robust methods for handling LLM outputs that deviate from expected formats (e.g., JSON parsing) could be explored.
  • Multimodal Tasks: Extending the framework to non-text modalities like image classification (via vision transformers) is a promising direction for future research.
  • Imperfect Ground Truths: The authors acknowledge that both the API labels and market data are imperfect ground truths, highlighting a general challenge in LLM validation. Future work could explore using expert-annotated datasets or more sophisticated external criteria.

Summary updated

BibTeX
@misc{megahed2025reliable,
  author = {Fadel M. Megahed and Ying-Ju Chen and L. Allision Jones-Farmer and Younghwa Lee and Jiawei Brooke Wang and Inez Maria Zwetsloot},
  title = {{Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applications}},
  year = {2025},
  eprint = {2505.14918},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.14918},
}
Figure