TL;DR
This paper introduces a novel Bayesian framework to directly sample and map the implicit "perceptual priors" of Vision-Language Models (VLMs). By pairing a controllable image generator with a Gibbs sampler judged by the VLM itself, the authors uncover deep-seated, covert demographic biases and conceptual associations that standard direct prompting fails to surface.
What Problem Does This Paper Address?
When Vision-Language Models make decisions, their outputs are heavily shaped by implicit representations and expectations about the world—their perceptual priors. Traditional interpretability research typically studies these models by holding inputs fixed and observing the outputs. However, this approach fails to map out the massive, high-dimensional space of what a model expects or prefers.
Furthermore, simply asking a model about its preferences (direct prompting) often fails because models are trained to output safe, aligned responses that mask their underlying biases. To truly understand and audit these models, researchers need a way to map their full, implicit prior distributions directly within their native visual modality.
Methods
The authors propose a Metropolis-Hastings within Gibbs sampler with Barker acceptance to let a VLM's own choices map out its internal priors.
The pipeline consists of three main components:
- Controllable Generative Model: They use FLUX.1-schnell combined with SliderSpace to create a latent space with independent, interpretable, and orthogonal visual "sliders" (e.g., smiling, feminine, eyeglasses, or texture).
- The VLM as a Judge: Instead of asking the VLM to manually adjust a slider (which causes oscillation and trial-and-error loops), the algorithm proposes a random change along a single slider dimension. The VLM is then presented with the current image and the proposed image in a randomized order.
- Barker Acceptance Step: The VLM answers a simple binary question (e.g., "Who looks more trustworthy?"). This binary choice acts as a mathematically rigorous acceptance step, allowing the system to iteratively sample and reconstruct the VLM's full posterior distribution across multiple dimensions.
Datasets
Because this method dynamically generates stimuli, it does not rely on standard static evaluation datasets. Instead, the authors generated and utilized the following:
- SliderSpace Training Data: To train the orthogonal latent sliders, they used 50,000 PCA samples over CLIP embeddings generated from diverse, domain-specific text prompts across four high-stakes domains: Faces, Affordances, Aesthetics, and Authenticity.
- Models Probed: The framework was evaluated using four frontier VLMs: Claude Sonnet 4.6, Gemini 3 Flash, GPT-5.4, and Qwen3-VL 235B.
- Ground-Truth Validation: To validate the sampler, they ran a low-dimensional experiment using a flat HSL color space targeting 8 human-recognizable color names (e.g., "sunset", "grass", "lavender").
Key Insights and Ideas
- Bypassing Alignment Filters: The Gibbs sampling approach successfully bypassed standard safety alignment filters to expose covert demographic biases. When judged on positive traits like Attractive, Trustworthy, Intelligent, and Hardworking, all four frontier models consistently pushed sliders associated with "Masculine/Receding Hairline" and "East Asian" below the neutral baseline.
- Universal Visual Archetypes: The models demonstrated strong consensus on abstract human traits. "Eyeglasses" emerged as the universal proxy for cognitive-positive traits (Intelligent, Hardworking, Serious), while "Smiling" dominated affective-positive traits (Fun, Trustworthy).
- Consensus Correlates with Visual Clarity: Models achieved high rank agreement on concepts with sharp, unambiguous visual signatures (such as identifying Manipulated vs. Authentic images). Conversely, agreement was lowest for diffuse, subjective concepts like Cheap or Experimental aesthetics.
Key Results
The large-scale evaluation spanned over 2.5 million trials and 1 billion tokens across 23 target concepts.
- Model Personas: Gemini 3 Flash exhibited the most extreme and polarized priors (e.g., pushing the Masculine/Receding Hairline slider to a high median of 0.87 for Criminal). Claude Sonnet 4.6 was consistently the least extreme and frequently out of step with the consensus of the other three models.
- Method Superiority: The authors compared their implicit Gibbs sampler against explicit baselines (like asking the model to rate a pre-generated sweep of images from 1 to 10). The Gibbs sampler recovered significantly more concentrated and informative posteriors, proving that explicit rating tasks fail to capture how models evaluate complex feature interactions.
Limitations and Future Work
- Generative Bottleneck: The framework's expressiveness is entirely bound by the underlying generative model. If a specific demographic, object, or concept cannot be rendered by FLUX or captured by SliderSpace, it remains completely invisible to the sampling chain.
- Axis Entanglement: Although SliderSpace enforces semantic orthogonality, complex multi-attribute interactions can still occur, potentially coupling unrelated visual factors.
- Lack of Causal Attribution: While the method successfully surfaces what a model's priors are, it cannot determine why they exist. The authors note that downstream causal analysis is required to untangle whether these biases originate from pre-training data, post-training RLHF alignment, or in-context prompt artifacts.
Summary updated