Prefer to Classify: Improving Text Classifiers via Auxiliary Preference Learning

Jaehyung Kim, Jinwoo Shin, Dongyeop Kang

TL;DR

Researchers have developed a novel framework called Prefer-to-Classify (P2C) that boosts text classifier performance by teaching models to recognize relative preferences between pairs of inputs. By learning not just that a sentence is "positive," but that it is more positive than another, models achieve significantly better accuracy and calibration across a wide range of NLP tasks.

What Problem Does This Paper Address?

Deep learning success in NLP relies heavily on large, human-annotated benchmarks. However, we are hitting a wall: collecting new input-output pairs is increasingly expensive and yields diminishing returns in accuracy.

Standard benchmarks use "instance-wise" labels—assigning a single category (like "Positive" or "Negative") to a single text. This approach ignores the nuanced, fine-grained information inherent in the data. For example, two movie reviews might both be labeled "Positive," but one is clearly more enthusiastic than the other. The authors argue that this relative, "pair-wise" information is a powerful, underutilized training signal that can help models generalize better, especially on "hard" examples where human annotators often disagree.

Methods

The authors propose the Prefer-to-Classify (P2C) framework, a multi-task learning approach that trains a model to perform classification and preference prediction simultaneously.

Figure 1. (a) Example of a pair-wise preference signal in the sentiment classification. (b) Auxiliary preference learning makes the classifier capture the fine-grained task information; e.g., predictions of the classifier become more aligned with human annotations. Test samples are divided into Hard, Normal, and Easy based on the annotators’ disagreement. (c) Improvement from the collected preference and P2C in various aspects, e.g., better accuracy and calibration. More results are presented in Section 4.4.

Key technical innovations include:

  • Multi-Preference Heads: Instead of one, the model uses multiple preference heads with a diversity regularization loss. This prevents the heads from collapsing into identical predictions, ensuring the model learns from diverse signals.
  • Consistency Regularization: A novel loss function that explicitly forces the model to have higher classification confidence in the "preferred" sample of a pair.
  • Informative Pair Sampling: Since training on all possible pairs ($N^2$) is computationally impossible, they introduced schemes to select the most "informative" pairs based on model disagreement or inconsistency.
  • Three Sources of Preference:
    1. Extractive: Derived for free from existing annotation records (e.g., if 4/5 annotators say "Positive" for text A and 3/5 for text B, A is preferred).
  • Generative: Obtained by querying Large Language Models (LLMs) like GPT-3 to compare two sentences.
  • Subjective: Collected directly from human crowd workers.

Datasets

The researchers evaluated P2C on ten diverse text classification datasets, including:

  • Standard Benchmarks: CoLA (linguistic acceptability), SMS Spam, Hate Speech, and Emotion.
  • Datasets with Annotation Records: DynaSent (R1 & R2), Stanford Politeness (Wiki & StackExchange domains), Offensive agreement, and MultiNLI.
  • Cross-Domain: The SUN Attribute dataset was used to demonstrate that P2C also works for image classification.

Key Insights and Ideas

  • Relative > Absolute: Pairwise comparisons provide a "calibration" signal that instance-wise labels lack. It helps the model map out the internal structure of a class.
  • Helping where it hurts: Preference learning provides the biggest gains on "Hard" samples—those ambiguous cases where humans struggle to agree on a single label.
  • LLMs as Cheap Teachers: Generative preferences from GPT-3 were found to be highly effective—sometimes even more so than adding more standard task labels—offering a cost-effective way to "evolve" existing benchmarks.
  • Better Calibration: Models trained with P2C aren't just more accurate; they are better calibrated, meaning their confidence scores more accurately reflect their actual likelihood of being correct.
Comparison of P2C results showing (a) pairwise signals, (b) better alignment with human nuances, and (c) overall improvements in accuracy and Expected Calibration Error (ECE).

Key Results

  • Generative Preference: P2C using GPT-3 signals achieved an 11.55% relative test error reduction on average compared to standard training.
  • Extractive Preference: Even when using "free" data from old annotation records, P2C outperformed state-of-the-art disagreement learning methods with a 4.27% relative error reduction.
  • Subjective Preference: Human-provided preferences yielded the highest gains, particularly in model calibration, reducing Expected Calibration Error (ECE) significantly compared to models trained only on task labels.
  • Beyond Text: In image classification (SUN Attribute), P2C achieved a 6.66% relative error reduction, proving the method's versatility.

Limitations and Future Work

The authors note that while extractive preference is "free," it requires raw annotation records which aren't always published with datasets. Subjective preference, while most accurate, remains expensive. Additionally, the $N^2$ nature of pairs requires careful sampling to remain efficient.

Future work could involve extending P2C to more complex NLP tasks beyond classification, such as sequence labeling or generative tasks, and further refining how LLMs can be used to generate high-quality auxiliary signals.

Summary updated

Abstract

The development of largely human-annotated benchmarks has driven the success of deep neural networks in various NLP tasks. To enhance the effectiveness of existing benchmarks, collecting new additional input-output pairs is often too costly and challenging, particularly considering their marginal impact on improving the current model accuracy. Instead, additional or complementary annotations on the existing input texts in the benchmarks can be preferable as an efficient way to pay the additional human cost. In this paper, we investigate task-specific preferences between pairs of input texts as a new alternative way for such auxiliary data annotation. From 'pair-wise' comparisons with respect to the task, the auxiliary preference learning enables the model to learn an additional informative training signal that cannot be captured with 'instance-wise' task labels. To this end, we propose a novel multi-task learning framework, called prefer-to-classify (P2C), which can enjoy the cooperative effect of learning both the given classification task and the auxiliary preferences. Here, we provide three different ways to collect preference signals in practice: (a) implicitly extracting from annotation records (for free, but often unavailable), (b) collecting explicitly from crowd workers (high paid), or (c) pre-trained large language models such as GPT-3 (low paid). Given existing classification NLP benchmarks, we demonstrate that the proposed auxiliary preference learning via P2C on them is effective in improving text classifiers. Our codes are publicly available.
BibTeX
@inproceedings{kim2023prefer,
  author = {Jaehyung Kim and Jinwoo Shin and Dongyeop Kang},
  title = {{Prefer to Classify: Improving Text Classifiers via Auxiliary Preference Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=SeNmPk-hVi7},
}
Figure