When evaluating a machine learning model, we typically measure how well it maps a specific input to a correct output. However, deep neural networks often harbor strong internal preferences—priors learned from skewed training distributions or induced by the formatting of a prompt. To see these preferences clearly, we need to strip away the signal.
Imagine asking a language model to classify the sentiment of the string "N/A", or asking a computer vision model to categorize a solid black square. Because these inputs contain zero task-relevant information, a perfectly unbiased model should output a uniform probability distribution across all possible classes. Any deviation from this uniformity exposes the model's intrinsic bias.
This technique, known as null-input or content-free probing, acts as a diagnostic baseline. By observing how a model responds to a blank slate, researchers can isolate internal parameters from external context, providing a powerful mechanism to audit fairness, optimize prompts, and calibrate predictions.
Why Null-Input Probing Works
To understand the intuition behind this approach, consider the standard formulation of conditional probability in a classification task: $P(Y \mid X)$, where $X$ is the input and $Y$ is the predicted label. In an ideal scenario, the model's prediction is driven entirely by the semantic features present in $X$. However, in practice, the model computes a distribution heavily influenced by its training history and prompt structure:
$$P(Y \mid X, \text{Prompt}, \theta)$$
where $\theta$ represents the frozen weights of the model.
By substituting $X$ with a completely uninformative or null token ($X_{\text{null}}$), such as an empty string, a placeholder, or a neutral tensor, we effectively eliminate the input features from the equation. Because $X_{\text{null}}$ contains no mutual information with any target class $Y$, the resulting distribution isolates the combined influence of the prompt's structure and the model's internal class priors:
$$P(Y \mid X_{\text{null}}, \text{Prompt}, \theta)$$
As pioneered by Shi Feng et al. in Calibrate Before Use: Improving Few-shot Performance of Language Models, this distribution can be used to calculate corrective parameters. If a model shows an inherent 70% preference for the "Positive" label when fed "N/A", we can construct a calibration matrix to shift the decision boundaries. This establishes a fair baseline for zero-shot and few-shot inference, significantly stabilizing performance without requiring a labeled validation set.
[Standard Input] "I love this movie!" ----> [ Model ] ----> P(Positive) = 0.95
[Content-Free Input] "N/A" ----> [ Model ] ----> P(Positive) = 0.70 <-- Intrinsic Bias Isolated!
Diagnosing and Calibrating Prompt Sensitivity
In-context learning (ICL) is notoriously sensitive to the choice and ordering of examples. Null-input probing provides a data-free metric to optimize these configurations. For instance, Huan Ma et al. developed a framework called G-fair-Prompting in Fairness-guided Few-shot Prompting for Large Language Models, which measures how biased a prompt template is toward certain labels when given a meaningless input string. By searching for example orderings that minimize the divergence from a uniform distribution on null inputs, they maximize downstream accuracy. Similarly, Zhichao Xu et al. leverage null inputs in In-Context Example Ordering Guided by Label Distributions to guide example ordering by evaluating the model's baseline label distribution.
However, even supposedly neutral prompts can carry hidden semantic associations. To address this, Kang He et al. introduced IntrCal in Prompt-Based Bias Calibration for Better Zero/Few-Shot Learning of Language Models, which uses an LLM to generate truly non-informative strings to calibrate internal bias parameters. To isolate fundamental model sensitivities from prompt-induced patterns, Sheng-Lun Wei demonstrates in Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models that zero-shot evaluation is critical for measuring a model's raw sensitivity to option permutations.
Auditing Societal and Cultural Biases
Beyond performance calibration, null-input probing serves as a powerful tool for auditing fairness and uncovering implicit societal prejudices. When a model is forced to generate text or fill in blanks from a neutral starting point, it relies heavily on its internal stereotypes.
Xiangjue Dong et al. utilize conditional probing with neutral stimuli in Probing Explicit and Implicit Gender Bias through LLM Conditional Text Generation to show how models inject gendered information into open-ended completions without explicit triggers. In the multimodal domain, Mohammad Nadeem et al. apply prompt-based bias auditing in Gender Bias in Text-to-Video Generation Models: A case study of Sora, using semantically neutral text inputs to reveal systematic demographic associations in generated video media.
This concept extends to cultural and socioeconomic defaults. Christabel Acquaye et al. use an unspecified context baseline in Susu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between Ghana and the US to show that a model's "neutral" reasoning state heavily defaults to Western cultural norms. Similarly, Ruixi Lin et al. substitute task-specific information with null placeholders in Mind the Biases: Quantifying Cognitive Biases in Language Model Prompting to quantify cognitive biases like availability bias and framing effects. When internal weights are completely inaccessible, Khyati Khandelwal et al. show in Indian-BhED: A Dataset for Measuring India-Centric Biases in Large Language Models that prompting a model to fill in missing information across neutral templates can successfully estimate localized social biases via majority voting.
Extending to Vision and Latent Representations
The principle of featureless input probing is equally potent in computer vision and deep representation learning. In long-tailed or class-imbalanced scenarios, networks often develop a strong bias toward majority classes.
To measure this, Hyuck Lee et al. introduce a method in CDMAD: Class-Distribution-Mismatch-Aware Debiasing for Class-Imbalanced Semi-Supervised Learning that evaluates a vision model on a solid color image; because the input lacks semantic features, any class preference directly reflects the internal prior distribution. Similarly, Yue Cheng et al. utilize a task-irrelevant baseline image (such as a solid black square) in Learning Dynamics of Logits Debiasing for Long-Tailed Semi-Supervised Learning as a proxy to track and prune majority-class bias during training. For generative models, Mike Laszkiewicz et al. isolate demographic biases in Benchmarking the Fairness of Image Upsampling Methods by feeding models inputs so heavily degraded that all informative signals are destroyed, forcing the model to rely purely on its learned phenotypic priors.
Taking this a step further, Wenjie Pan et al. propose a training paradigm in Fair Training with Zero Inputs where the model is explicitly trained on a tensor of zeros. By regularizing the model to produce a uniform output on a blank input, they successfully mitigate reliance on spurious background correlations and class imbalances.
Finally, null probing can operate directly on internal vector spaces or generation speeds. Ziyang Xu et al. demonstrate in Take Care of Your Prompt Bias! Investigating and Mitigating Prompt Bias in Factual Knowledge Extraction that subtracting the hidden layer vector representation of a "blank" prompt from a real query vector strips away prompt-induced bias. In reasoning models, Renfei Dang et al. extract latent priors in The First Impression Problem: Internal Bias Triggers Overthinking in Reasoning Models by forcing models to skip intermediate reasoning steps, revealing the immediate "gut feeling" that can trigger computational overthinking.
Key Foundational Work
Calibrate Before Use: Improving Few-shot Performance of Language Models (2021) — contributed 4 concept elements to this direction.
Fairness-guided Few-shot Prompting for Large Language Models (2023) — contributed 1 concept elements to this direction.
Mind the Biases: Quantifying Cognitive Biases in Language Model Prompting (2023) — contributed 1 concept elements to this direction.