Explainable Text Classification with LLMs: Enhancing Performance through Dialectical Prompting and Explanation-Guided Training

Huaming Du, Lei Yuan, Cancan Feng, Guisong Liu, Gang Kou, Carl YangEMNLP 2025

TL;DR

This paper introduces Dialectical Explanation Training (DET), a novel method that enhances text classification performance and interpretability by using Large Language Models (LLMs) to generate multi-perspective explanations. These explanations then guide the training of smaller, more efficient models, while also incorporating uncertainty estimation for robust decision-making.

What Problem Does This Paper Address?

Traditional text classification models, while accurate, often lack interpretability, making it hard to understand why a prediction was made. Conversely, LLMs can provide explanations, but they still lag behind fine-tuned smaller models in pure classification accuracy and can be prone to overconfidence and hallucinations. Existing explanation methods often rely on keyword extraction, which can miss crucial contextual nuances. This paper aims to bridge this gap by developing a method that combines the strengths of both LLMs (for rich explanations) and smaller models (for efficiency and accuracy), while also addressing the critical need for understanding model uncertainty, especially in high-stakes domains like finance and law.

Methods

The core of the proposed method, Dialectical Explanation Training (DET), consists of two main steps:

  1. Dialectical Prompting: This novel prompting strategy uses LLMs to generate explanations for all potential labels of a given text. Instead of a single explanation, it encourages the LLM to explore multiple perspectives, considering context-dependent expressions and complex linguistic phenomena. This is achieved through a carefully crafted prompt that instructs the LLM to analyze the input text and provide detailed justifications for each possible category, ensuring no interpretation is omitted.
  2. Explanation-Guided Training: The dialectical explanations generated in the first step are then used as features to train a smaller classification model (e.g., DeBERTa-base). This approach leverages the rich contextual information from the LLM's explanations to improve the smaller model's predictive power and interpretability. Additionally, the paper incorporates the theory of Evidential Deep Learning (EDL) to quantify the uncertainty of the model's predictions, allowing for more reliable decision-making.

Datasets

The authors evaluated their method on seven representative datasets across various domains:

  • StockEmotions: Finance (sentiment analysis)
  • Overruling: Law (judicial opinions)
  • Stance4Trump: Politics (stance classification)
  • Ethos: Hate speech detection
  • RTE: Natural Language Inference
  • ValueNet: Value alignment
  • AGNews: News classification
  • MASSIVE: Multilingual natural language understanding

Key Insights and Ideas

  • Dialectical Explanations are Key: The core idea is that by forcing an LLM to consider all possible interpretations and provide justifications for each, the resulting explanations become richer, more nuanced, and better at capturing complex linguistic phenomena than single-perspective explanations.
  • Small Models + Big Explanations = Win-Win: Training smaller, efficient models with LLM-generated explanations offers a powerful combination, achieving high accuracy and interpretability without the computational cost of running large LLMs for every prediction.
  • Uncertainty Matters: Incorporating Evidential Deep Learning is crucial for high-stakes applications. Knowing when a model is uncertain allows for rejecting predictions that might be unreliable, enhancing trust and safety.
  • Context is King: The paper strongly emphasizes that traditional keyword-based explanations often fail because they miss the broader context, a problem DET aims to solve by generating more comprehensive, context-aware rationales.

Key Results

  • DET significantly outperforms baselines in both classification performance (ACC and F1 scores) and explanation quality (EQ scores) across most of the tested datasets.
  • On the RTE dataset, DET achieved an 11.80% improvement in ACC, a 13.25% improvement in F1, and a remarkable 133% increase in EQ compared to the second-best baseline.
  • The custom Explanation Quality (EQ) metric, evaluated by advanced LLMs (GPT-40 and Claude 3.5-sonnet), showed high correlation with human expert evaluations, validating its reliability.
  • DET demonstrated stronger robustness to out-of-distribution data compared to baseline models, indicating better domain adaptation capabilities.
  • The uncertainty analysis showed that correctly classified samples generally have lower uncertainty, and higher uncertainty is observed for misclassified samples, aligning with expectations.

Limitations and Future Work

  • Data Dependency: DET still requires labeled data for training, which might be a limitation in scenarios where labels are scarce or unavailable.
  • LLM Fine-tuning: The effectiveness of fine-tuning LLMs directly with dialectical explanations remains an open question for further exploration.
  • Generalization: The generalization ability of DET needs further improvement.
  • Future Work: The authors suggest incorporating causal inference or stable learning to address generalization issues and explore the application of DET to LLM fine-tuning. They also propose using dialectical prompting selectively for ambiguous cases to reduce computational cost.

Summary updated

BibTeX
@inproceedings{du2025explainable,
  author = {Huaming Du and Lei Yuan and Cancan Feng and Guisong Liu and Gang Kou and Carl Yang},
  title = {{Explainable Text Classification with LLMs: Enhancing Performance through Dialectical Prompting and Explanation-Guided Training}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure