Abstract
Zero-shot text classification (ZSTC) using large-scale language models offers high flexibility but suffers from significant computational overhead. This is primarily due to the linear scaling of inference time with the number of candidate labels. We present a framework that utilizes Conformal Prediction (CP) to prune the label space using a computationally inexpensive base classifier. By generating a mathematically guaranteed prediction set, the framework reduces the workload of the primary "heavy" model. Experimental results across five benchmarks demonstrate that this approach reduces inference time by up to 25% and prunes the label space by over 41%, all while maintaining a 99% accuracy guarantee relative to the original zero-shot model.
Introduction
Modern zero-shot classification typically leverages Natural Language Inference (NLI) or Next Sentence Prediction (NSP) architectures. These models utilize "cross-encoders," which process the input text and a candidate label together to produce a relevance score. While accurate, this architecture creates a linear bottleneck: for a dataset with $N$ categories, the model must perform $N$ computationally expensive forward passes per input. In high-cardinality datasets like HWU64 (64 labels), this scaling makes real-time deployment and large-scale processing prohibitively expensive and environmentally taxing.
This paper addresses the efficiency gap by introducing a two-stage filtering mechanism. The goal is to discard obvious label mismatches using a low-cost method without sacrificing the accuracy of the high-performance "slow" model.
Methodology
The proposed framework consists of a three-step pipeline designed to balance speed and precision:
- The Base Classifier (The Filter): A computationally "cheap" model generates initial scores for all possible labels. The authors evaluated four variants:
CP-Token(token overlap),CP-Glove(static embedding similarity),CP-CLS(a distilled BERT model), andCP-Distil(a parameter-efficient NLI model). - Conformal Mapping: Using the principles of Conformal Prediction, the scores from the base classifier are transformed into a prediction set. Unlike standard classifiers that output a single "top-1" prediction, CP generates a set of labels guaranteed to contain the ground-truth label (as defined by the slow model) with a user-specified probability (e.g., 99%).
- The Zero-shot Model (The Ranker): The expensive, high-accuracy model (such as BART-large or BERT-base) only processes the labels contained within the pruned CP prediction set.
Self-Supervised Calibration
A significant hurdle in zero-shot settings is the absence of labeled data for calibration. To overcome this, the authors employ a self-supervised calibration strategy. The heavy zero-shot model is used to label a small set of unlabeled data from the target domain. These "silver" labels are then used to calibrate the CP framework, ensuring the fast filter's logic is mathematically aligned with the slow model's decision boundaries.
Experimental Evaluation
The framework was evaluated on five diverse benchmarks covering intent and topic classification:
- Intent Classification: SNIPS (7 labels), ATIS (17 labels), and HWU64 (64 labels).
- Topic Classification: AG’s News (4 labels) and Yahoo! Answers (10 labels).
The primary models used for the "slow" ranking stage were BART-large (NLI-based) and BERT-base (NSP-based).
Key Results and Insights
Efficiency Gains
The framework achieved substantial performance improvements across all tested configurations:
- Inference Speedup: Average inference time was reduced by 25.6% for NLI-based models and 22.2% for NSP-based models.
- Label Pruning: The number of labels the heavy model had to process dropped by an average of 41.09% (NLI) and 43.38% (NSP).
- Scalability: The most significant benefits were observed in the HWU64 dataset, confirming that the method's utility increases as the label space grows.
Accuracy and Denoising
By design, the system maintains a 99% accuracy rate relative to the original model. Interestingly, in certain instances, the final system's accuracy actually improved after filtering. This suggests that the CP filter acts as a denoiser, removing irrelevant or confusing labels that might have led the larger model to a false positive.
Filter Selection
While more complex filters like CP-CLS produced smaller, more refined label sets, the simplest method—CP-Token—often yielded the best overall speedup. This is because the computational cost of the CP-Token filter is negligible, allowing the majority of the time savings to be realized in the ranking stage.
Conclusion
This research demonstrates that the efficiency-accuracy tradeoff in zero-shot classification is negotiable. By using Conformal Prediction as a mathematical safety net, developers can significantly reduce the carbon footprint and latency of LLM-based classifiers. Future work may extend this framework to auto-regressive models like GPT-4, where the cost of generating multiple verbalizers is even higher, and explore automated methods for selecting the optimal base classifier for specific domains.
Summary updated