Abstract
Prompt-based learning is susceptible to intrinsic bias present in pre-trained language models (LMs), leading to sub-optimal performance in prompt-based zero/few-shot settings. In this work, we propose a null-input prompting method to calibrate intrinsic bias encoded in pre-trained LMs. Different from prior efforts that address intrinsic bias primarily for social fairness and often involve excessive computational cost, our objective is to explore enhancing LMs' performance in downstream zero/few-shot learning while emphasizing the efficiency of intrinsic bias calibration. Specifically, we leverage a diverse set of auto-selected null-meaning inputs generated from GPT-4 to probe intrinsic bias of pre-trained LMs. Utilizing the bias-reflected probability distribution, we formulate a distribution disparity loss for bias calibration, where we exclusively update bias parameters ($0.1\%$ of total parameters) of LMs towards equal probability distribution. Experimental results show that the calibration promotes an equitable starting point for LMs while preserving language modeling abilities. Across a wide range of datasets, including sentiment analysis and topic classification, our method significantly improves zero/few-shot learning performance of LMs for both in-context learning and prompt-based fine-tuning (on average $9\%$ and $2\%$, respectively).
Introduction
Pre-trained language models (LMs) are inherently "opinionated," harboring intrinsic biases inherited from their massive training corpora. These biases—such as a preference for common words or specific sentiments—often lead to sub-optimal performance in prompt-based learning. For instance, when prompted to classify sentiment, a model might favor a "positive" label simply because the token is more frequent in its training data, rather than based on the input text.
Existing solutions, often termed "Output Calibration," typically re-weight final probabilities. However, these are often "band-aid" fixes that do not modify the model itself, struggle with task-specific fine-tuning, and rely on human-crafted null strings (like "N/A") that may not effectively probe the model's internal state.
Methodology: Intrinsic Bias Calibration (IntrCal)
To address these issues, we introduce IntrCal, a method to "re-center" LMs by calibrating their intrinsic biases using auto-generated "null-meaning" inputs. The IntrCal pipeline consists of three primary stages:
- Auto-Generating Null Inputs: Rather than a single "N/A" string, we utilize GPT-4 to generate a diverse set of 1,000 null-meaning inputs, ranging from random symbols to content-free sentences.
- NSP Filtering: To ensure these inputs integrate well with task prompts, we use a BERT-large model to calculate Next Sentence Prediction (NSP) scores. We prioritize null inputs that logically precede the task's answer format, ensuring the model's attention mechanism is properly engaged.
- Parameter-Efficient Calibration: A "fair" model should produce a uniform probability distribution across labels when given a null input. We freeze the model's main weights and update only the bias parameters ($B_{LM}$), which constitute approximately 0.1% of the total weights. This efficiency prevents "catastrophic forgetting" of general language knowledge.
Experimental Setup
IntrCal was evaluated across eight diverse English classification datasets, covering topic classification (AGNews, DBPedia, TREC), sentiment analysis (Subj, SST-5), and aspect-level sentiment (Laptop, Restaurant, and Twitter reviews). The method was tested on various architectures, including RoBERTa-large, GPT-2 XL, and Llama-2 (7B).
Key Insights and Results
Our experiments revealed several critical findings:
- The "Bias" is in the Bias: Updating only additive bias terms is sufficient to counteract intrinsic bias, proving more efficient and safer than full-model fine-tuning.
- The "One-Batch" Heuristic: For zero-shot tasks, we discovered that calibrating for just one batch of null inputs provides a consistent performance peak. Beyond this, the model may start producing purely uniform, and thus uninformative, predictions.
- Performance Gains: On RoBERTa-large, IntrCal improved zero-shot in-context learning by an average of 9% over the baseline and provided a 2% boost in few-shot fine-tuning settings.
- Preserving Language Ability: Pseudo-perplexity tests on WikiText confirmed that this light-touch calibration does not degrade the model's fundamental language understanding.
Conclusion and Future Work
While IntrCal significantly improves classification performance, several avenues for future research remain. The current study focuses on classification; extending this to generative tasks or regression is a logical next step. Additionally, while the "one-batch" rule is a robust heuristic, more rigorous unsupervised methods for determining optimal stopping points could further refine the calibration process. Finally, complex tasks like Natural Language Inference (NLI) may require more sophisticated null-input structures to align with sentence-pair formats.
Summary updated