TL;DR
Cross-encoder models for zero-shot text classification achieve high accuracy but suffer from poor efficiency because their computational cost scales linearly with the number of candidate labels. This paper introduces a framework that uses a fast base classifier combined with a Conformal Predictor (CP) to filter out unlikely labels with strict statistical guarantees. By pruning the label space first, the authors reduce average inference time by up to 25.6% without compromising the classification performance.
What Problem Does This Paper Address?
Modern zero-shot text classification often relies on formulating the task as Natural Language Inference (NLI) or Next Sentence Prediction (NSP). For instance, to classify a tweet into a category, a large pre-trained language model (PLM) evaluates pairs like (text, "This text is about sports") using a cross-encoder architecture.
While highly accurate, cross-encoders require full self-attention over the combined text and hypothesis. This means the model must perform a separate forward pass for every single candidate label. If a dataset has 64 possible labels, the model runs 64 times for one input sequence. This linear scaling makes deployment computationally expensive, slow, and environmentally unfriendly. The authors aim to break this linear dependency by quickly filtering out irrelevant labels before the heavy cross-encoder model runs.
Methods
The authors propose using Conformal Prediction (CP) to safely reduce the label space. Unlike standard classifiers that output a single label, a conformal predictor outputs a set of candidate labels that is guaranteed to contain the correct label with a user-defined probability (e.g., $1 - \alpha$, where $\alpha$ is the error rate, set to 1% in this work).
The workflow operates in two stages:
- Label Filtering (Fast): A lightweight "base classifier" computes a non-conformity score (a measure of how poorly a label fits the text). Labels with scores below a calculated threshold ($\hat{q}$) are kept; the rest are discarded.
- Final Prediction (Accurate): The expensive zero-shot cross-encoder model processes only the small subset of remaining candidate labels.
Calibration Without Human Labels
Typically, conformal prediction requires a human-labeled calibration dataset to find the threshold $\hat{q}$. In a true zero-shot setting, such data doesn't exist. To bypass this, the authors use the large zero-shot model itself to label an unlabeled calibration set. Because the goal is simply to avoid discarding the label that the large model would have chosen, calibrating on the model's own predictions provides valid coverage guarantees.
The authors experiment with four base classifiers of varying complexity:
- CP-Token: Counts token overlap between the text and representative words for each class (lowest complexity).
- CP-Glove: Computes cosine similarity using static GloVe embeddings.
- CP-CLS: A task-specific
distilbert-basemodel fine-tuned on the zero-shot model's predictions. - CP-Distil: A smaller, parameter-efficient pre-trained NLI model (
distilroberta-base-nli).
Datasets
The framework was evaluated on five distinct datasets:
- Intent Classification: SNIPS (7 labels), ATIS (17 labels), and HWU64 (64 labels).
- Topic Classification: AG's News (4 labels) and Yahoo! Answers (10 labels).
The primary zero-shot baselines used for evaluation were bart-large-mnli (for NLI) and bert-base-uncased (for NSP).
Key Insights and Ideas
- The Complexity-Size Trade-off: Stronger base classifiers (like CP-CLS or CP-Distil) generate much smaller candidate label sets (Average Prediction Set size, or APS). However, they take longer to compute. Conversely, simpler methods like CP-Token are incredibly fast but yield larger sets.
- Pruning Can Boost Performance: Surprisingly, filtering labels with CP occasionally increased final classification accuracy. This suggests that narrowing down the label space removes noisy or confusing candidate labels that might otherwise trick the large PLM.
- Scale Matters Most: The framework provides the highest efficiency gains on datasets with large label spaces (like HWU64) or when the downstream zero-shot model is exceptionally large. For small label spaces, the overhead of the base classifier can negate the speedup.
Key Results
- Inference Speedup: The framework reduced average inference time by 25.6% for NLI-based models and 22.2% for NSP-based models.
- Label Reduction: The average number of target labels evaluated per sample dropped by 41.09% (NLI) and 43.38% (NSP).
- Performance Preservation: All four base classifiers achieved valid empirical coverage matching the nominal coverage at low error rates ($\alpha = 0.01$), meaning accuracy did not drop below the predefined 1% threshold.
Limitations and Future Work
- Language and Cultural Bias: The evaluation was restricted entirely to English-language datasets, meaning the findings may not immediately generalize to multilingual settings or other cultural contexts.
- Inherent Model Biases: The framework relies on the predictions of the large zero-shot model for calibration. Any underlying gender, age, or social biases present in the original PLM will be carried over or potentially amplified by the filtering process.
- Future Directions: The authors plan to explore more sophisticated non-conformity scores to further shrink the average prediction set size and investigate applying this framework to prompt-based auto-regressive models.
Summary updated