Neuroscientists have long worried about seed bias: if you decide in advance which brain region or which stimulus categories to study ("faces," "places"), you can only confirm what you expected. Recent tools promise a way out. They fit a model that predicts each brain location's fMRI response from an image, then use large pretrained image and language models to describe what each location prefers. One version writes a caption; another generates an image. These tools have reported new findings, such as sub-regions of body-selective cortex split by social versus object context, and a food region split into processed food versus produce. Our claim is that this approach does not remove the experimenter's hypothesis. It moves the hypothesis into the pretrained model. The images in the fMRI dataset used by these tools come from MS COCO, a public image set very likely present, along with its captions and near-duplicates, in the web-scale data those models were trained on. The model's memory of COCO can therefore act as a hidden seed. It decides which concepts can be named and which splits look clean.
We propose to measure this directly. We will run the published pipelines on inputs whose true tuning we control: brain-like signals with planted preferences, and "nonsense" signals with no real tuning. We will also run them with pretrained models whose exposure to COCO we manipulate. The deliverable has two parts. The first is a quantitative audit of how much of each reported discovery survives once the pretrained model's prior is controlled. The second is a validation protocol, independent of the pretrained model, that future brain-mapping work can adopt. The project borrows its logic from the language-model contamination literature, which found that high benchmark scores can reflect memory rather than skill. Here we ask whether a clean cortical "discovery" can reflect the describing model's memory rather than the brain.
Core Claim
Generative brain-mapping replaces seed bias with prior bias, and benchmark contamination makes that prior bias worse. Concretely, our hypothesis is that when brain-derived tuning points toward concepts that are rare in the pretraining data, the pipeline will relabel them as the nearest common COCO-like concept. We also expect noise-level brain signals to still yield confident, category-coherent captions and clean sub-cluster splits. A further consequence follows. The validation these papers use, zero-shot classification by CLIP (an image–text model) of the generated outputs, is scored by the same model family that generated them, so its high agreement cannot rule out prior-driven artifacts.
The motivating papers are these. BrainSCUBA maps each voxel's linear tuning into CLIP embedding space and projects it onto a bank of 2 million natural-image embeddings from LAION-A and Open Images. It then captions the result with a GPT-2-based captioner, and it acknowledges that biases of CLIP and GPT-2 "may be reflected in the brain captions." BrainDiVE steers Stable Diffusion 2.1 with gradients from a CLIP-based brain encoder, but it still targets predefined regions. Neither paper quantifies prior leakage, and that gap is what we address. Multigranular Evaluation for Brain Visual Decoding shows that generative decoders hallucinate details supplied by their priors, such as grass around a giraffe. It does so for reconstruction of seen images, not for selectivity discovery, and its scorer is itself a multimodal model that could be contaminated. We study the discovery setting and require a scorer that does not depend on the backbone.
The contamination literature constrains which designs can work. The Benchmark Data Contamination survey reports contamination rates of 1–45% on question-answering benchmarks. It also distinguishes semantic-level exposure from verbatim exposure, and string matching cannot catch the former. On the Fragility of Benchmark Contamination Detection in Reasoning Models shows that loss-based membership detectors fall to roughly 50–58% AUROC after some training regimes. Taken together, these results argue against detecting contamination in CLIP and in favor of intervening on it. ABCD supplies the method for our null test. Its NonsenseQA benchmark of meaningless questions exposed models scoring above 95% from format alone, and we build a brain-side analogue.
Research Program
The first component is a planted-tuning simulation. Using the Natural Scenes Dataset images (about 10,000 COCO images per subject), we synthesize voxel responses whose tuning is defined outside CLIP. The tuning functions come from COCO object annotations, from hand-specified visual attributes, and from features of a vision model trained without web text. We add noise matched to each real voxel's measured reliability. Some planted tunings target concepts that are frequent in captions, such as "person" and "food." Others target concepts that are rare or hard to put into words, such as specific textures or object conjunctions; BrainSCUBA flags non-verbalizable preferences as a limitation. We then run the full BrainSCUBA and BrainDiVE pipelines on these synthetic voxels. Recovery is scored against the planted ground truth, which tells us when the pipeline renames, merges, or invents structure.
The second component is a null-brain test. We fit encoders to NSD responses after shuffling the image–response pairing within each subject, which preserves noise level and spatial smoothness. This is the brain-side NonsenseQA. We record how often the pipeline produces sub-cluster splits and confident category captions with no real signal behind them. This gives a false-discovery rate for "sub-specializations" that the published work does not report.
The third component manipulates exposure. We train matched small image–text encoders and captioners on two corpora of equal size and composition that differ only in whether COCO images, captions, and their near-duplicates are included. Near-duplicates are found by embedding similarity, not n-grams. Separately, we swap BrainSCUBA's 2-million-image projection bank for banks with and without COCO-near neighbors. If the discovered splits for real brains change with exposure while held-out encoding accuracy stays constant, the prior is shaping the discovery.
The fourth component is validation that does not depend on the backbone. For each reported discovery, we turn its claim into a prediction about held-out NSD trials: indoor-preferring voxels of the occipital place area should respond more to indoor images, and the "social group" cluster of the extrastriate body area should respond more to multi-person scenes. We label those images with COCO's human annotations rather than with CLIP. A discovery counts as brain-attributable if this held-out effect exceeds its shuffled-null distribution, survives every exposure condition, and is recovered in the matching planted-tuning simulation.
Baselines
The naive reference point is the published pipelines validated as their authors validated them, by zero-shot CLIP classification of generated outputs. BrainSCUBA reports 97% semantic category alignment for food regions in subject S2, and BrainDiVE reports alignment "up to 100% for some subjects" (BrainSCUBA, BrainDiVE). We will rerun both on the same NSD subjects with the same encoder (CLIP ViT-B/16 with a linear adapter). The question is not whether we beat these numbers. It is whether they stay high on null voxels; if they do, the metric is uninformative. No reported result found on Lacuna for either pipeline under shuffled or planted-tuning inputs.
The strongest existing competitor is a standard permutation null combined with top-activating natural images. In this method, each voxel's best-driving NSD images are inspected directly and compared against shuffled-response nulls. It needs no generative prior to propose candidates, which makes it the obvious skeptic's tool. We give it the same subjects, the same voxel selection, and the same held-out split. Our protocol beats it if it separates planted from absent tuning more accurately on simulated voxels, scored by recovery AUROC against ground truth. It must also flag at least one discovery that the permutation-plus-top-images check passes but that fails under exposure manipulation or misnames planted rare concepts.
The third baseline is a decomposition with no pretrained backbone. Following the hypothesis-free voxel decomposition of Murty et al., which found a food-selective component that category-based studies had missed, we decompose the measured NSD responses directly. This tests whether the structure in the brain data exists before any pretrained model names it. Agreement between those components and the BrainSCUBA clusters counts as evidence that the clusters are real.
Two ablations isolate the key ingredient. One runs the planted-tuning simulation with a single public backbone and no exposure manipulation. The other runs exposure manipulation on real data with no simulation. If either alone accounts for the audit's conclusions, the combined design is unnecessary.
Why not just…?
Why not just run a permutation test on the shuffled data? It is part of our design, but it only answers whether some structure exceeds chance. It cannot tell whether real but weak brain signal was renamed by the prior, for example rare texture tuning captioned as "food." The ABCD finding that format alone yields above 95% on nonsense questions (ABCD) suggests that prior-driven coherence can survive with real signal present. The planted-tuning comparison against the permutation baseline will show whether this concern matters in practice. If the permutation test catches every simulated failure, we will report that the cheaper check suffices.
Why not just deduplicate COCO from the backbone's training data? Retraining public CLIP or Stable Diffusion at full scale is not feasible. Deduplication by string or hash also misses semantic-level exposure, which the contamination survey separates from verbatim leakage. EvoEval's 39.4% drop under paraphrase, reported there, shows how much depends on that kind of familiarity. Our matched small-scale corpora are the controlled version of this alternative. Their limit is that they are smaller than the real backbones, and we will report that limit plainly.
Why not just validate discoveries with new fMRI experiments? Collecting new data is the gold standard, and we do not claim to replace it. But it costs scanner time for every candidate discovery. Our backbone-independent held-out check and the null false-discovery rate are meant to decide which discoveries justify that cost, and they apply to data that already exist.
Why not just use backbone-free decomposition for everything? Decomposition finds components but does not name them. The case for generative tools is that they propose interpretations. We use decomposition as a reference point rather than a replacement. A serious threat to this project would be if decomposition plus manual inspection matched the generative pipelines on every simulated case; that outcome would be reported as a negative result for the generative approach.
Decisive Checks
The project succeeds if at least one of the following holds. First, shuffled-response voxels yield sub-cluster splits and CLIP-validated captions at a non-trivial rate. Second, planted rare concepts are systematically renamed as frequent COCO-like concepts while planted frequent concepts are recovered. Third, a published sub-specialization, such as the EBA social-versus-object split or the OPA indoor-versus-outdoor split, changes under COCO-exposure manipulation or fails the annotation-based held-out test. If none of these occurs, so that discoveries are stable across exposure, pass the backbone-independent held-out tests, and null voxels yield nothing, the result is still valuable. It would give the first evidence that these generative tools escape both seed bias and prior bias, and the audit protocol would become a standard pre-registration check for future generative brain-mapping claims.
References
- BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity
- Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models
- Multigranular Evaluation for Brain Visual Decoding
- Benchmark Data Contamination of Large Language Models: A Survey
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- ABCD: All Biases Come Disguised
- A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition