TL;DR
Researchers have developed a novel framework called Prefer-to-Classify (P2C) that boosts text classifier performance by teaching models to recognize relative preferences between pairs of inputs. By learning not just that a sentence is "positive," but that it is more positive than another, models achieve significantly better accuracy and calibration across a wide range of NLP tasks.
What Problem Does This Paper Address?
Deep learning success in NLP relies heavily on large, human-annotated benchmarks. However, we are hitting a wall: collecting new input-output pairs is increasingly expensive and yields diminishing returns in accuracy.
Standard benchmarks use "instance-wise" labels—assigning a single category (like "Positive" or "Negative") to a single text. This approach ignores the nuanced, fine-grained information inherent in the data. For example, two movie reviews might both be labeled "Positive," but one is clearly more enthusiastic than the other. The authors argue that this relative, "pair-wise" information is a powerful, underutilized training signal that can help models generalize better, especially on "hard" examples where human annotators often disagree.
Methods
The authors propose the Prefer-to-Classify (P2C) framework, a multi-task learning approach that trains a model to perform classification and preference prediction simultaneously.
Key technical innovations include:
- Multi-Preference Heads: Instead of one, the model uses multiple preference heads with a diversity regularization loss. This prevents the heads from collapsing into identical predictions, ensuring the model learns from diverse signals.
- Consistency Regularization: A novel loss function that explicitly forces the model to have higher classification confidence in the "preferred" sample of a pair.
- Informative Pair Sampling: Since training on all possible pairs ($N^2$) is computationally impossible, they introduced schemes to select the most "informative" pairs based on model disagreement or inconsistency.
- Three Sources of Preference:
- Extractive: Derived for free from existing annotation records (e.g., if 4/5 annotators say "Positive" for text A and 3/5 for text B, A is preferred).
- Generative: Obtained by querying Large Language Models (LLMs) like GPT-3 to compare two sentences.
- Subjective: Collected directly from human crowd workers.
Datasets
The researchers evaluated P2C on ten diverse text classification datasets, including:
- Standard Benchmarks: CoLA (linguistic acceptability), SMS Spam, Hate Speech, and Emotion.
- Datasets with Annotation Records: DynaSent (R1 & R2), Stanford Politeness (Wiki & StackExchange domains), Offensive agreement, and MultiNLI.
- Cross-Domain: The SUN Attribute dataset was used to demonstrate that P2C also works for image classification.
Key Insights and Ideas
- Relative > Absolute: Pairwise comparisons provide a "calibration" signal that instance-wise labels lack. It helps the model map out the internal structure of a class.
- Helping where it hurts: Preference learning provides the biggest gains on "Hard" samples—those ambiguous cases where humans struggle to agree on a single label.
- LLMs as Cheap Teachers: Generative preferences from GPT-3 were found to be highly effective—sometimes even more so than adding more standard task labels—offering a cost-effective way to "evolve" existing benchmarks.
- Better Calibration: Models trained with P2C aren't just more accurate; they are better calibrated, meaning their confidence scores more accurately reflect their actual likelihood of being correct.
Key Results
- Generative Preference: P2C using GPT-3 signals achieved an 11.55% relative test error reduction on average compared to standard training.
- Extractive Preference: Even when using "free" data from old annotation records, P2C outperformed state-of-the-art disagreement learning methods with a 4.27% relative error reduction.
- Subjective Preference: Human-provided preferences yielded the highest gains, particularly in model calibration, reducing Expected Calibration Error (ECE) significantly compared to models trained only on task labels.
- Beyond Text: In image classification (SUN Attribute), P2C achieved a 6.66% relative error reduction, proving the method's versatility.
Limitations and Future Work
The authors note that while extractive preference is "free," it requires raw annotation records which aren't always published with datasets. Subjective preference, while most accurate, remains expensive. Additionally, the $N^2$ nature of pairs requires careful sampling to remain efficient.
Future work could involve extending P2C to more complex NLP tasks beyond classification, such as sequence labeling or generative tasks, and further refining how LLMs can be used to generate high-quality auxiliary signals.
Summary updated