Abstract
Masked language models like BERT can perform text classification in a zero-shot fashion by reformulating downstream tasks as text infilling. However, this approach is highly sensitive to the template used to prompt the model, yet practitioners are blind when designing them in strict zero-shot settings. In this paper, we propose an alternative mining-based approach for zero-shot learning. Instead of prompting language models, we use regular expressions to mine labeled examples from unlabeled corpora, which can optionally be filtered through prompting, and used to finetune a pretrained model. Our method is more flexible and interpretable than prompting, and outperforms it on a wide range of tasks when using comparable templates. Our results suggest that the success of prompting can partly be explained by the model being exposed to similar examples during pretraining, which can be directly retrieved through regular expressions.
Introduction
Zero-shot learning with Large Language Models (LLMs) has traditionally relied on prompting, where a task is reformulated as a "fill-in-the-blank" problem. For instance, sentiment analysis might be framed as: "The movie was [MASK]." While powerful, prompting is notoriously brittle. Minor variations in the template—such as changing "It was [MASK]" to "The film was [MASK]"—can result in significant performance fluctuations.
In a true zero-shot scenario, practitioners lack the labeled data necessary to validate which prompt is most effective, effectively designing templates in the dark. Furthermore, prompting often suffers from "surface form competition," where the model struggles to distinguish between synonymous labels. This paper argues that the efficacy of prompting stems from the model encountering similar patterns during pre-training. To exploit this, the authors propose bypassing the "black box" of prompting by explicitly retrieving these examples from large corpora to use for supervised fine-tuning.
Methodology: Zero-Shot Mining
The proposed approach, termed Zero-Shot Mining, follows a three-step pipeline to generate and utilize weakly-labeled data:
- Mining: Standard prompting templates are converted into regular expressions (regex). For example, a sentiment prompt like
{INPUT}. It was {VERBALIZER}is transformed into a search pattern:(is|was) {good|great|bad|awful}. {INPUT}. These patterns are then used to scan massive unlabeled text corpora (such as C4) to find sentences that naturally follow these label-indicative phrases. - Filtering (Optional): To mitigate noise in the mined data, a standard zero-shot prompt is used to "double-check" the extracted examples. If the prompt's prediction contradicts the mined label with high confidence, the example is discarded.
- Fine-tuning: The resulting weakly-labeled dataset is used to fine-tune a standard pretrained model (e.g., RoBERTa) with a dedicated classification head.
Unlike traditional prompting, which typically evaluates one verbalizer at a time, this method naturally aggregates multiple patterns and synonyms into a single, diverse training set, providing greater flexibility and interpretability.
Experimental Results
The authors evaluated their method against standard benchmarks including sentiment analysis (Amazon, IMDb, SST-2), topic classification (AG News, DBPedia), and Natural Language Inference (MNLI, RTE). The source corpus for mining was the first 100 shards of the C4 dataset.
As shown in the results table, the mining-based approach significantly outperformed standard prompting across all categories:
- Overall Performance: The proposed method achieved a macro-average accuracy of 72.0%, an 8.8-point improvement over multi-verbalizer prompting (63.2%) and a 15.4-point jump over single-verb prompting (56.6%).
- Topic Classification: The gains were most pronounced here. On AG News, accuracy soared from 34.0% with prompting to 79.2% with mining.
- NLI: Even on complex logical tasks, mining maintained a lead, averaging 53.0% compared to 47.5% for prompting.
- Robustness: The method proved less sensitive to the specific choice of verbalizers, as it aggregates signals from a wide array of retrieved patterns.
Discussion and Future Work
The core insight of this work is that prompting may be viewed as a convoluted form of pattern retrieval. By making this retrieval explicit through regex, the process becomes more controllable and interpretable. While mined data is inherently noisy—for instance, a negative pattern might capture a sarcastic positive sentence—the sheer volume of data allows the model to learn a robust signal. Furthermore, explicitly fine-tuning a classification head proves more robust than relying on internal next-token probabilities.
However, limitations remain. Crafting regex patterns for complex tasks like multiple-choice QA or deep linguistic nuance (e.g., the Word-in-Context benchmark) is challenging. Additionally, the method incurs computational overhead, requiring several hours for corpus searching and fine-tuning. Future research may explore more sophisticated retrieval mechanisms, such as semantic search, and advanced filtering techniques to better handle the noise inherent in web-scale data.
Summary updated