Prefer to Classify: Improving Text Classifiers via Auxiliary Preference Learning

Jaehyung Kim, Jinwoo Shin, Dongyeop Kang

Introduction

Standard text classification typically follows an "instance-wise" paradigm: a model examines a single input and assigns it a label from a predefined set. While effective, this approach overlooks the comparative nature of human judgment. Humans naturally evaluate information through the lens of relative preference—recognizing, for instance, that one product review is more emphatically positive than another, even if both are labeled as "Positive."

The "Prefer-to-Classify" (P2C) framework proposes that teaching models to perform these relative comparisons alongside standard classification can significantly enhance their performance, reliability, and understanding of task nuances. By integrating auxiliary preference learning, P2C extracts more signal from existing data and cheap external sources like Large Language Models (LLMs) to improve the underlying classification task.

The P2C Framework

P2C is a multi-task learning framework designed to augment a standard Transformer-based classifier (such as RoBERTa). The model is trained on two primary objectives: predicting the correct class for individual samples and predicting which of two samples more strongly represents a specific label.

Technical Components

  1. Multi-Head Preference Learning: The architecture utilizes a shared backbone with multiple "preference heads." To prevent these heads from converging on identical features, a diversity loss ($L_{div}$) is applied, forcing the model to capture diverse facets of the preference signal.
  2. Consistency Regularization ($L_{cons}$): This component acts as a bridge between classification and preference tasks. It enforces a logical constraint: if the model determines that Text A is "more" representative of a class than Text B, its classification confidence for Text A must be higher than for Text B.
  3. Informative Pair Sampling: To manage the $N^2$ complexity of pairwise comparisons, the authors employ disagreement-based and inconsistency-based sampling. This focuses training on the most ambiguous and informative pairs, maximizing learning efficiency.

Sources of Preference Signals

A key advantage of P2C is its flexibility in acquiring preference data. The authors explore three distinct avenues for obtaining these comparative signals:

  • Generative Preferences: Utilizing LLMs like GPT-3 to generate pairwise comparisons. This proves to be a cost-effective alternative to manual labeling, often providing more value than simply adding more standard task labels.
  • Extractive Preferences: Mining "hidden" signals from existing datasets. By looking at raw annotator records (e.g., the distribution of votes among multiple humans), the framework identifies which samples are more prototypical of a label based on consensus levels. This effectively turns discarded annotator disagreement into a "free" training signal.
  • Subjective Preferences: Direct human ranking of pairs, providing the highest quality signal, though at a higher collection cost.

Experimental Results

The researchers evaluated P2C across diverse benchmarks, including linguistic acceptability (CoLA), sentiment analysis (DynaSent), and hate speech detection. To demonstrate generalizability, the method was also successfully tested on the SUN Attribute image dataset.

Performance Gains

The implementation of P2C led to a 11.55% relative reduction in test error when using generative preferences compared to standard training. Furthermore, when utilizing extractive preferences from raw annotation records, P2C outperformed existing state-of-the-art methods for learning from annotator disagreement by 4.27%.

Model Calibration and Robustness

Beyond accuracy, P2C significantly improves model calibration. A well-calibrated model's predicted probabilities align closely with actual likelihoods, making it more trustworthy for real-world deployment. The framework also showed its greatest strengths on "hard" samples—those where human annotators originally struggled to reach a consensus—suggesting that preference learning helps models navigate boundary cases more effectively than instance-wise learning alone.

Conclusion

The "Prefer-to-Classify" framework demonstrates that relative comparisons are a powerful auxiliary signal for NLP. By leveraging "free" data from annotator disagreements or cheap comparisons from LLMs, P2C builds classifiers that are not only more accurate but also better calibrated. Future research may extend this comparative approach to generative tasks like summarization or dialogue, where absolute "ground truth" is often harder to define than in classification.

Summary updated

BibTeX
@inproceedings{kim2023prefer,
  author = {Jaehyung Kim and Jinwoo Shin and Dongyeop Kang},
  title = {{Prefer to Classify: Improving Text Classifiers via Auxiliary Preference Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=SeNmPk-hVi7},
}
Figure