Executive Summary
The Prefer-to-Classify (P2C) framework improves text classification by teaching models not just to label individual samples, but to compare them. By leveraging "preference signals"—determining which of two texts more strongly represents a category—researchers achieved up to an 11.55% relative error reduction and significantly improved model calibration across various NLP benchmarks.
The Limits of Instance-Wise Labels
Deep learning success in NLP is largely built on massive, human-annotated benchmarks. However, these benchmarks typically rely on "instance-wise" labels: a single category assigned to a single text (e.g., "Is this review positive or negative?").
This approach leaves valuable information on the table. Instance-wise labels struggle to capture fine-grained nuances, especially for "hard" samples where human annotators might disagree. While collecting more data is the traditional solution, it is prohibitively expensive. Instead, the authors ask: Can we better exploit existing data by looking at the relationship between samples?
By shifting the focus to pairwise comparisons (e.g., "Which of these two positive reviews is more positive?"), models can learn a much richer, more continuous representation of the task.
The Prefer-to-Classify (P2C) Framework
To harness this relational data, the authors introduce Prefer-to-Classify (P2C), a multi-task learning framework that trains a classifier and a "preference predictor" simultaneously.
How P2C Works
- Multi-Task Learning: The model shares a Transformer backbone (like RoBERTa) but branches into two types of heads: a standard classification head and multiple preference heads.
- Diverse Preference Heads: To capture different nuances of "preference," the authors use multiple heads and a diversity regularization loss. This prevents the heads from simply mimicking each other, forcing them to find different informative signals in the data.
- Consistency Regularization: This is the core "bridge" between the two tasks. It enforces a simple intuition: if the model prefers Text A over Text B for a "Positive" sentiment task, the classification head should show higher confidence for Text A than for Text B.
- Informative Pair Sampling: Since comparing every possible pair of texts is computationally impossible ($O(N^2)$), the authors developed sampling schemes to pick the most "confusing" or "disagreed-upon" pairs to train on.
Sources of Preference
The paper identifies three ways to obtain these pairwise signals:
- Extractive: "Free" labels harvested from existing annotation records (e.g., if 5/5 people voted Text A as positive, but only 3/5 voted for Text B, Text A is "preferred").
- Generative: "Low-cost" labels generated by prompting Large Language Models (LLMs) like GPT-3 to compare pairs.
- Subjective: "High-quality" labels collected directly from human crowd-workers.
Key Insights and Results
The researchers evaluated P2C across ten diverse text classification datasets, including standard benchmarks (CoLA, Hate Speech) and annotation-rich datasets (DynaSent, MultiNLI). They even tested the framework on the SUN Attribute dataset (image classification) to prove the method isn't limited to text.
- Preferences as a "Soft" Signal: Preference learning acts as a form of implicit label smoothing. It provides a gradient of intensity that a binary "0 or 1" label cannot provide.
- Boosting "Hard" Samples: The biggest gains were seen in "Hard" samples—those where human annotators originally disagreed. The pairwise comparison helps the model navigate the ambiguity that trips up standard classifiers.
- Better Calibration: Beyond accuracy, P2C models are better "calibrated." When the model says it is 80% sure, it is actually right about 80% of the time, making the model more trustworthy for real-world deployment (reflected in lower Expected Calibration Error).
- LLMs as Teachers: Interestingly, P2C with GPT-3 preferences outperformed GPT-3's own zero-shot classification performance. This suggests that LLMs are better at comparing two things than labeling one thing in isolation.
Performance Highlights: * Using generative preferences (GPT-3), P2C achieved an average 11.55% relative test error reduction compared to vanilla training. * When using extractive preferences, P2C outperformed existing state-of-the-art methods for learning from annotator disagreement by 4.27%. * Qualitative results showed that P2C predictions were more closely aligned with the distribution of human votes than any other baseline.
Limitations and Future Work
While powerful, P2C has a few hurdles. Extractive preferences require raw annotation records, which many older or smaller datasets do not provide. Furthermore, even with smart sampling, training on pairs is more computationally intensive than training on single instances.
Overall, Prefer to Classify offers a compelling argument for moving beyond the "one-label-per-item" paradigm, showing that teaching models to compare is a highly efficient way to squeeze more intelligence out of existing benchmarks.
Summary updated