Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Models

Mozes van de Kar, Mengzhou Xia, Danqi Chen, Mikel Artetxe

Abstract

Masked language models like BERT can perform text classification in a zero-shot fashion by reformulating downstream tasks as text infilling. However, this approach is highly sensitive to the template used to prompt the model, yet practitioners are blind when designing them in strict zero-shot settings. In this paper, we propose an alternative mining-based approach for zero-shot learning. Instead of prompting language models, we use regular expressions to mine labeled examples from unlabeled corpora, which can optionally be filtered through prompting, and used to finetune a pretrained model. Our method is more flexible and interpretable than prompting, and outperforms it on a wide range of tasks when using comparable templates. Our results suggest that the success of prompting can partly be explained by the model being exposed to similar examples during pretraining, which can be directly retrieved through regular expressions.

Introduction

Zero-shot learning via prompting has emerged as a powerful paradigm for utilizing large-scale language models without task-specific training data. By framing a classification task as a cloze-style completion (e.g., "The movie was [MASK]" for sentiment analysis), practitioners can leverage the internal knowledge of models like BERT or RoBERTa.

However, this approach suffers from a significant "blind spot": performance is extremely sensitive to the specific wording of the prompt and the choice of verbalizers (the words used to represent labels). In a true zero-shot setting, where no labeled validation set exists, researchers cannot objectively determine which prompt is most effective. This research proposes that the success of prompting is actually a form of implicit pattern matching against the model's pretraining data. By shifting from prompting to explicit mining, we can create a more robust and interpretable zero-shot pipeline.

Methodology: The Zero-Shot Mining Pipeline

The proposed method, Zero-Shot Mining, replaces the "black box" of prompting with a three-step "mine-then-finetune" workflow:

  1. Mining with Regular Expressions: Using the same verbalizers typically used in prompting (e.g., "excellent," "terrible"), the authors define simple regular expressions to search a massive unlabeled corpus (the C4 dataset). For instance, to find positive sentiment examples, a regex might look for patterns like (is|was) excellent. {INPUT}, where {INPUT} captures the preceding text to serve as a training example.
  2. Optional Filtering: To mitigate the noise inherent in regex mining, the authors optionally use zero-shot prompting to verify the mined labels. If the prompt's prediction disagrees with the mined label and the prompt is highly confident, the example is discarded. This ensures a higher quality of weak supervision.
  3. Finetuning: A standard pretrained model (RoBERTa-base) is then finetuned on this mined, weakly-labeled dataset using a traditional classification head.

Unlike standard prompting, which usually relies on a single template, this mining approach allows for the aggregation of diverse patterns and dozens of different verbalizers into a single, comprehensive training set.

Experimental Setup

The authors evaluated the mining approach against standard zero-shot prompting across 12 diverse datasets:

  • Sentiment Analysis: Amazon, IMDb, MR, SST-2, and Yelp.
  • Topic Classification: AG News, DBPedia, and Yahoo Topics.
  • Natural Language Inference (NLI): MNLI, QNLI, RTE, and SNLI.

The mining source was the first 100 shards of the C4 corpus, representing approximately 9.8% of the total dataset.

Key Results and Discussion

Performance Gains

The mining-based method significantly outperformed standard zero-shot prompting, achieving an average improvement of 8.8 percentage points across all 12 tasks. The most dramatic improvement occurred in topic classification; on the AG News dataset, standard prompting achieved only 34.0% accuracy, while the mining approach reached 79.2%.

Robustness and Interpretability

One of the primary advantages of mining is its stability. While prompting performance can fluctuate wildly based on minor template changes, mining aggregates multiple patterns, making it less susceptible to the "prompt engineering" lottery. Furthermore, the method is highly interpretable; practitioners can inspect the mined examples to understand exactly what the model is learning. Qualitative analysis revealed that even though approximately 40% of mined sentiment examples were neutral or noisy, the sheer volume of data provided enough signal for the model to generalize effectively.

Efficiency

While the mining and finetuning process requires additional computation (roughly 2–7 hours on a single GPU), the resulting model is highly efficient. At inference time, it has the same or lower computational cost as a prompted model, as it uses a standard classification head rather than a masked language modeling head.

Conclusion and Future Work

This research suggests that the knowledge required for zero-shot tasks is already present in the pretraining distribution and can be more effectively extracted through explicit mining than through prompting. By treating zero-shot learning as a data retrieval and weak supervision problem, we gain both performance and transparency.

Future work remains in refining the pattern engineering process for more complex logical tasks and developing more sophisticated filtering mechanisms to handle the noise in mined data. However, the current results provide a compelling alternative to the instability of prompt-based learning.

Summary updated

Abstract

Masked language models like BERT can perform text classification in a zero-shot fashion by reformulating downstream tasks as text infilling. However, this approach is highly sensitive to the template used to prompt the model, yet practitioners are blind when designing them in strict zero-shot settings. In this paper, we propose an alternative mining-based approach for zero-shot learning. Instead of prompting language models, we use regular expressions to mine labeled examples from unlabeled corpora, which can optionally be filtered through prompting, and used to finetune a pretrained model. Our method is more flexible and interpretable than prompting, and outperforms it on a wide range of tasks when using comparable templates. Our results suggest that the success of prompting can partly be explained by the model being exposed to similar examples during pretraining, which can be directly retrieved through regular expressions.
BibTeX
@inproceedings{kar2022don,
  author = {Mozes van de Kar and Mengzhou Xia and Danqi Chen and Mikel Artetxe},
  title = {{Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=1iYev7F8g2},
}
Figure