Abstract
Large Language Models (LLMs) have demonstrated remarkable zero-shot capabilities across various NLP tasks. However, their performance in text classification significantly degrades as the number of candidate categories increases. This paper identifies two primary obstacles: "boundary ambiguity" among semantically similar labels and "inherent bias" toward specific positions or tokens. To address these, we propose a two-stage "Reduce-then-Compare" framework. This approach first narrows the label space through self-reduction and then employs a novel Pairwise Contrastive Chain-of-Thought (PC-CoT) to refine the final classification. Empirical results show that our framework improves accuracy by over 50% for models like GPT-3.5-turbo on complex benchmarks.
The Challenge of Large-Scale Classification
While LLMs excel at simple tasks, their classification performance is fragile when faced with dozens or hundreds of potential labels. The authors pinpoint two fundamental bottlenecks that emerge as the label space expands:
- Boundary Ambiguity: When multiple options share semantic similarities (e.g., "top_up_failed" vs. "pending_top_up"), LLMs struggle to establish clear decision boundaries, leading to confusion between related intents.
- Inherent Bias: Models often exhibit "position bias," favoring options at the start of a list, and "token bias," an unexplained preference for specific words (e.g., "movies" or "convert") regardless of the input context.
Simply increasing model parameters or context windows does not mitigate these issues; even 70B parameter models are susceptible to distraction by a high volume of options.
The Reduce-then-Compare Framework
To overcome these limitations, the researchers introduce a two-stage methodology designed to simplify the decision-making process for the LLM.
Stage 1: Self-Reduction
Instead of a single-step selection from a massive list, the model first reduces the candidate pool to a manageable set (typically the top 5). Two techniques are utilized:
- Iterative Top Reduction (ITR): The model progressively identifies and isolates the most probable candidates.
- Cluster-Based Window Reduction (CBWR): Options are grouped by similarity, and the model selects from "windows" containing representatives from different clusters to ensure diverse candidates are considered.
Stage 2: Pairwise Contrastive Comparison (PC-CoT)
The narrowed list undergoes a "tournament" of pairwise comparisons. To ensure robustness, the authors developed PC-CoT, a reasoning chain that mirrors human logic:
- Similarity Analysis: The model identifies commonalities between two labels.
- Difference Analysis: It explicitly defines the "key element" distinguishing them.
- Decision Making: A final choice is made based on this contrastive analysis.
Key Insights: The Impact of Option Volume
A critical finding is that LLMs are highly effective at binary choices but struggle as the label count grows. For instance, GPT-3.5-turbo maintains ~94% accuracy in binary classification but drops significantly as more options are introduced.
The "noise" generated by numerous options triggers internal biases. In open-source models like LLaMA2, moving the correct answer to the first position can alter performance by over 35%, highlighting the severity of position bias.
Experimental Results
The framework was evaluated on four challenging benchmarks: Banking77, HWU64, LIU54, and Clinic150.
- Performance Gains: GPT-3.5-turbo saw a 54.1% relative improvement in average accuracy compared to standard zero-shot classification.
- Effective Debiasing: The method reduced token bias in LLaMA2-70B-Chat, yielding a 36.88% relative performance boost.
- Scalability: Benefits were consistent across model sizes, from 1B to 70B parameters.
Conclusion and Future Work
The "Reduce-then-Compare" framework effectively mitigates the inherent weaknesses of LLMs in large-scale classification. While the iterative nature increases computational cost, it provides a significant accuracy advantage over standard prompting. Future research will focus on optimizing inference efficiency and exploring the framework's performance with specialized long-context models.
Summary updated