How Many Bits of a Movie Does Each Brain Region Need?

Models that predict fMRI responses to movies are usually given the full, uncompressed video. They predict brain activity by passing frames through a large vision network and fitting a linear map from its features to each brain region. This proposal turns video compression into an experimental knob. We compress the movie at a controlled number of bits per pixel before feature extraction, then trace how prediction accuracy falls in each cortical region as bits are removed. We do this with two kinds of compressor. Pixel codecs (H.265, VVC) throw away fine texture first. Machine-oriented "semantic" codecs, built to keep what a recognition network needs, throw away appearance and keep meaning. The shape of each region's accuracy-versus-bits curve then gives a quantitative profile, in a common unit (bits), of which stimulus information that region's predictable activity depends on.

The second half of the problem is the reason this is hard and worth doing carefully. Movie fMRI datasets scan only a handful of people for many hours, so any per-region quantity estimated from them is exposed to the winner's curse: noisy estimates that happen to cross a threshold come out inflated. We therefore treat the region-wise "bits needed" estimate as a statistic. We measure how much fMRI data it takes to estimate it reliably by subsampling scan hours and subjects within a deeply scanned dataset. We also test a specific, falsifiable side claim: when training data are scarce, a lower-rate input can predict high-level regions better than the full video, and the best rate rises as data grow. The two-sentence version is this. Compression lets us remove information from a movie in measured amounts and see which brain regions notice. Small fMRI samples make such measurements fragile, so we also establish how many hours of scanning they need.

Core Claim

Each cortical parcel has a characteristic rate–accuracy curve. Pixel and semantic codecs dissociate along the visual hierarchy. Early visual cortex should lose accuracy steeply under pixel codecs at low rates, and semantic codecs should not rescue it. Lateral occipital and superior temporal regions should keep most of their accuracy under semantic codecs at rates where pixel codecs have collapsed. We define, for parcel $p$ and codec family $c$, the critical rate

$$R_{p,c}(\tau) = \min \{ R : r_{p,c}(R) \ge \tau \, r_{p}(\infty) \},$$

where $r_{p,c}(R)$ is the held-out Pearson correlation between predicted and measured activity in parcel $p$ when the video is coded at rate $R$ (bits per pixel), $r_p(\infty)$ is the correlation with uncompressed video, and $\tau$ is a tolerance such as 0.95. The new object is the cortical map of $R_{p,c}$, together with its sampling distribution as a function of scan hours. Neither literature has produced this map.

Where This Sits

The motivation comes from a gap on both sides. Video coding for machines shows that compression artifacts damage machine perception in ways human viewers ignore: Does Video Compression Impact Tracking Accuracy? documents this for trackers. It also shows that sending semantic information instead of pixels recovers task accuracy at very low rates. A Coding Framework and Benchmark Towards Low-Bitrate Video Understanding (VCS) pairs a standard base codec with a learned semantic stream and reports over 50% bit savings on UCF101. Free-VSC aligns its compressed stream to frozen vision foundation models such as VideoMAE and DINOv2, reaching 62.24% HMDB51 accuracy at 0.02 bpp versus 34.35% for VVC. These works supply the method, in the form of the codecs we sweep. They evaluate only against machine benchmarks and never against cortex.

On the brain side, all movie encoding work uses uncompressed media. That includes the Algonauts 2025 winners, the stacked-regression entry, and the BOLD Moments study, which maps early TSM ResNet50 layers to V1 and deeper layers to LOC and STS. The encoding/decoding survey likewise notes that no one has examined compressed inputs. The closest connecting work, cross-subject semantic decoding from video fMRI, also assumes full-fidelity clips and addresses decoding rather than information requirements. Our proposal differs from all of these by manipulating stimulus information in measured bits and reading out region-wise sensitivity.

Two results constrain the design. Scaling laws for language encoding models in fMRI finds that encoding accuracy plateaus near 30B parameters despite cross-validated ridge. The authors hypothesize the cause is too few time points (about 33,000 per subject) for very wide features. They also find that ten times more data per subject improves accuracy by 122%, so any conclusion drawn from a few hours may change with more. The stacked-regression report finds that fine-tuning a vision backbone beyond 3 epochs overfits the small CNeuroMod cohort. That is why we keep all feature extractors frozen and use only regularized linear readouts. The general statistical concern comes from Statistical Generalizability in Neuroimaging and the grey-matter reproducibility assessment, which show that small-sample effect estimates inflate and fail to replicate. This is our reason for reporting critical rates with subsampling distributions rather than point estimates.

Research Program

The main dataset is CNeuroMod as used in the Algonauts 2025 challenge. It offers about 80 hours of movie viewing per subject in a small cohort, with 1,000 cortical parcels and an out-of-distribution test set. This depth is what makes the sample-size analysis possible: we can subsample training hours down to the regime of a typical small study and still evaluate on a fixed held-out set. BOLD Moments, with 10 subjects and 1,102 three-second clips, serves as a replication set. There, between-subject variability rather than scan hours is the limiting factor.

We sweep video rate from very low (about 0.01 bpp) to visually lossless with four codec families. The first is H.265. The second is VVC. The third is VCS, which lets us vary the split between base-layer pixel bits and semantic-stream bits at fixed total rate. The fourth is Free-VSC. If Free-VSC code is unavailable, we will reimplement it; we flag this as a risk and keep VCS as the semantic codec of record. Pixel codecs are decoded to frames and passed through a frozen VideoMAE encoder. Free-VSC's decoder emits features aligned to VideoMAE, so the feature space is shared across codecs. Audio and transcript features (Whisper and Llama, as in the winning systems) are held uncompressed and fixed throughout. Each curve therefore measures the marginal value of visual bits for a parcel, not the loss of the whole stimulus. Readouts are ridge regressions with lagged features and identical cross-validation.

For the sample-size arm, we refit every curve at 1, 3, 10, 30 hours and the full training set, with repeated random draws of runs. We report the bootstrap distribution of $R_{p,c}$ and its split-half reliability across draws. We also report the smallest training budget at which the parcel ranking of critical rates stabilizes. Separately, we record the rate that maximizes held-out accuracy at each budget, to test the claim that the best rate rises with data in high-level regions.

Baselines

The naive reference point is the uncompressed pipeline: the same frozen VideoMAE features plus fixed audio and text features, with the same ridge readout. This defines $r_p(\infty)$ and is the "do nothing" condition that every compressed point is compared against. For context, the strongest reported number on this benchmark is a mean Pearson r of 0.21455 across 1,000 parcels by the first-place TRIBE system (Insights from the Algonauts 2025 Winners). The fourth-place linear model came within 0.006 of it. Our deliberately simple pipeline should land near the range the stacked-regression report reports for single vision backbones (r of 0.161 for slow_r50, 0.178 after LoRA). If our uncompressed reference fell far below that, any curve shapes would be suspect. We do not aim to beat TRIBE, because the question is about relative curves, not leaderboard accuracy.

The strongest competitor for the scientific claim is layer-depth mapping in the style of BOLD Moments. Each parcel is assigned a preferred layer of the uncompressed network, and "texture versus semantics" is read from that depth. It is run on identical data, with the same encoder and readout, by fitting each layer separately and recording the best layer per parcel. Rate curves beat it only if critical-rate maps explain held-out structure that layer preference does not. Concretely, they must predict which parcels lose accuracy under a new, unseen codec better than layer preference does.

The key ablation is a matched-dimension control. Uncompressed features are reduced by PCA, and separately by random projection, to the effective dimensionality of each compressed condition, with the same ridge tuning grid. This isolates whether any low-rate advantage at small data comes from removing stimulus information selectively or merely from having fewer feature dimensions.

A second ablation replaces codecs with one-parameter degradations, spatial blur and frame-rate reduction, calibrated to the same file size. This tests whether learned bit allocation matters at all.

Why not just…?

Why not just read texture-versus-semantics off network layer depth, as BOLD Moments already does? Layer depth confounds abstraction with architecture. Every layer sees the full stimulus, so a parcel that prefers deep layers might still depend on fine texture. Compression removes information from the input while the network stays fixed, which is an intervention rather than a correlation. This is a real threat to the claim of added value. The baseline comparison above settles it: if layer preference predicts held-out codec sensitivity as well as critical-rate maps do, the rate sweep adds bookkeeping, not insight, and we will say so.

Why not just reduce feature dimensionality with PCA or stronger ridge to handle the small-sample problem? This could plausibly work as well for the side claim. That is why the side claim is secondary and is tested directly against PCA and random projection at matched dimension. Ridge alone did not remove the plateau in Scaling laws for language encoding models in fMRI, but PCA was not tested there. If PCA matches compression at every training budget, we conclude the small-data benefit is purely dimensional. The main contribution, the critical-rate map, does not depend on this outcome, because PCA cannot tell us which stimulus information a region needs.

Why not just blur or temporally subsample the frames, as in classic psychophysics? Blur is a single low-pass knob, so it cannot separate appearance from meaning. Semantic codecs keep object and action identity at rates where blurred video has lost it, as the HMDB51 gap between Free-VSC and VVC shows. Still, if bit-matched blur curves reproduce the pixel-codec curves, that is useful: it would mean commodity codecs behave like low-pass filters for the brain. The blur ablation is designed to reveal exactly this.

Why not just restore compressed frames before feature extraction? VCS reports that its semantic stream beats the restoration models BasicVSR++ and STDF on machine tasks. CDRE (Embedding Compression Distortion in Video Coding for Machines) reports beating PromptIR. Restoration also adds an uncontrolled generative prior that invents texture, which would corrupt the bit accounting. We include restoration only as an exploratory condition.

Decisive Checks

The first check is the dissociation itself. At bit rates of 0.05 bpp and below, the accuracy gap between semantic and pixel codecs should be positive in LOC and STS parcels. It should be near zero or negative in V1, and the difference between these regional gaps must hold on the out-of-distribution test films and replicate in BOLD Moments. If semantic codecs help or hurt all parcels uniformly, the region-specific claim fails.

The second check is reliability. Critical-rate maps should reach a split-half rank correlation above a pre-registered level by some training budget. Reporting that budget is a direct, practical answer to how much scanning such questions require. If the maps never stabilize even at full CNeuroMod depth, the measurement is not viable with current data, and that negative result is itself worth publishing.

The third check is the small-data side claim. In high-level parcels, the accuracy-maximizing rate should lie below full rate at 1–3 training hours, rise toward full rate as hours increase, and outperform matched-dimension PCA. Each of the three components can fail independently.

The fourth check compares brain and machine. We overlay each parcel's curve on the same codec's HMDB51 accuracy curve. This answers Gap 1's open question of whether cortical predictability degrades faster or slower than machine recognition as bits are removed.