Discrete Scene Codes as FiLM Switches for Frozen Encoders in Few-Demonstration Robot Learning

Robot imitation learning has to work from very few demonstrations, often 50 or fewer per task. Under that constraint it faces a familiar dilemma. A frozen, web-pretrained image encoder does not overfit, but it attends to the wrong things. Fine-tuning the encoder fixes the attention but memorizes the handful of demonstrations. This note proposes a middle path built from two simple pieces. The first is a small discrete "scene code": the observation is snapped to one of a few learned prototypes describing what kind of situation the robot is in. The second is feature-wise linear modulation (FiLM), a per-channel scale and shift applied to a network's internal features. Each discrete code switches on its own FiLM scale-and-shift for the frozen encoder. The encoder's continuous features still carry fine geometry and smooth motion to the action generator. Only a small table of per-code modulations is learned from the scarce demonstrations.

In two sentences: let a coarse, quantized reading of the scene decide how to reweight a frozen encoder's channels, and let the encoder's continuous features decide where exactly to move. The bet is that restricting the modulation to a finite set of learned affine transforms is the right amount of task adaptation when demonstrations are scarce. It should be more adaptive than a frozen encoder, less able to overfit than fine-tuning, and more robust to visual shift than modulation predicted from continuous features.

Core Claim

Consider an imitation policy with a frozen encoder in the low-data regime (5 to 50 demonstrations per task). We claim that predicting the encoder's FiLM parameters from a small number of discrete codes will beat three alternatives at matched parameter count and tuning budget: predicting them from continuous pooled features, fine-tuning the encoder, and leaving it frozen. The advantage should shrink as the number of demonstrations grows, and it should be largest under visual distribution shift. Both patterns would follow if the gain comes from the discrete bottleneck acting as a variance-reducing prior on the modulation, not from extra capacity.

The specific mechanism works as follows. Let $h_\ell$ be the channel activations of frozen encoder layer $\ell$. Let $c_1,\dots,c_M$ be $M$ codes, each drawn from a codebook of size $K$ and computed from an unmodulated early-layer pass. The modulated features are

$$\tilde h_\ell = \Big(1 + \sum_{m=1}^{M} G_\ell[c_m]\Big) \odot h_\ell + \sum_{m=1}^{M} B_\ell[c_m],$$

where $G_\ell$ and $B_\ell$ are learned lookup tables of per-channel scales and shifts, one row per code. The tables are initialized to zero, so training starts from the frozen encoder exactly. With $K=1$ this reduces to a single static task-level FiLM. As $K$ grows, the discrete codes approach a continuous FiLM generator. The codebook size is therefore a single knob that runs from "frozen-with-bias" to "fully adaptive," and the central prediction is an inverted-U in success rate against $K$ at small demonstration counts.

How This Relates to Prior Work

The motivation comes from Feature Extractor or Decision Maker. That paper shows frozen out-of-distribution encoders average about 43% success against about 85% for end-to-end training, because they attend to background clutter. Fine-tuning recovers success only from 26% to 61%. The paper calls for task-conditioned encoders but never tests FiLM on a frozen encoder. On the cost of adapting large backbones on small robot datasets, UniJEPA and Discrete Diffusion VLA both report that this erodes pretrained capabilities. UniToken reports that fine-tuning a continuous encoder alongside a codebook needs learning rates around $1\times10^{-5}$ to avoid collapse. These results are why our encoder stays frozen.

The FiLM mechanism is borrowed directly from Diffusion Policy. It conditions a denoising action generator on visual features through FiLM, and our policy head reuses it unchanged. Unpacking the Individual Components of Diffusion Policy shows FiLM reaching 64% on Adroit Relocate where concatenation collapses to nearly 2%, which justifies modulation over concatenation. Phase-Aware Policy Learning conditions FiLM on a hand-designed discrete phase clock and finds it beats mixture-of-experts switching. Our discrete codes can be read as a learned, vision-derived replacement for that phase clock. The differences are that our modulation acts on the perception encoder rather than the control network, and that nobody hand-labels the phases.

The hybrid-representation literature supplies the decomposition idea. HART and DC-AR split images into a discrete base and a continuous residual for generation. UniToken places discrete and continuous tracks side by side in one sequence and lets self-attention mix them. MemoryVLA pairs continuous perceptual tokens with discrete semantic gist tokens and fuses them with attention and gating. None of these lets the discrete part control the continuous part multiplicatively, and none is designed for few-demonstration control. In our design the discrete path never enters the action generator as content. It only reweights channels.

One result needs to be addressed head on. UniJEPA observes that discrete visual tokens "collapse or loop back" in policy feature spaces and lose temporal progression, so it abandons them. We accept that observation and design around it. Our codes are meant to be coarse and repetitive, recurring whenever a scene configuration recurs, and the continuous features remain the only carrier of progression. If UniJEPA's failure arose from asking discrete tokens to encode motion, a gating-only role should avoid it. A further check comes from How Clever Is the FiLM Model: FiLM sits at chance on spatial-relation questions without curriculum. Channel-wise scaling cannot express "left of." We therefore expect the least benefit on LIBERO-Spatial and treat that suite as a stress test, not a showcase.

Research Program

The codebook is learned without action labels, which is where data scarcity enters a second time. We run vector quantization on pooled early-layer features of the frozen encoder over frames from LIBERO-90 (the LIBERO pretraining suite), using images only. We use product quantization with $M$ codes of size $K$, and add a commitment loss and a mild temporal-consistency term so that neighboring frames tend to share codes. The tokenizer uses images only, never action supervision. This matters because LIBERO found that naive supervised pretraining on manipulation data can hurt downstream policies, and we want to keep the code-learning step clear of that effect.

The policy is then trained on the target tasks' scarce demonstrations. The encoder is frozen with the code-indexed FiLM tables inserted at three or four layers. A task-language embedding is added to the code embeddings, and the action head is the standard Diffusion Policy U-Net. Only the FiLM tables and the action head are trained. A second variant fine-tunes the codebook jointly, to test whether the action-free codes are already sufficient.

Three analyses accompany the success numbers. First, we sweep $K$ and $M$ against the number of demonstrations to map the inverted-U. Second, we align codes with annotated subtask boundaries on LIBERO-Long, which tests whether the codes behave like learned phases. Third, we compare saliency maps with the background-attention diagnostics of Visual Alignment Testing, to ask whether the modulation actually moves attention onto task-relevant regions.

Baselines

The naive reference point is Diffusion Policy on a frozen encoder with no modulation. It shows how much any adaptation buys. The VAT paper's roughly 43% frozen average is the expected magnitude of this failure in its own setting, though that number is not on our benchmarks.

The strongest competitor on the actual question is continuous-input FiLM: a small network maps the same pooled early-layer features straight to per-layer scales and shifts, with no quantization. The parameter count is matched to the lookup tables plus codebook. This baseline isolates the discrete bottleneck, the one ingredient we claim matters. To keep it from being a strawman, we give it its own sweep over weight decay, dropout, and width under the same tuning budget. We also run a version with a variational information bottleneck limited to roughly $M\log_2 K$ bits. That version separates "discreteness" from "few bits."

The second strong competitor is fine-tuning the encoder, both fully and with low-rank adapters, under the small learning rates UniToken found necessary. VAT reports that fine-tuning lifts OOD encoders from 26% to 61%.

The third competitor is attention-based fusion in the style of UniToken: the discrete code embeddings are appended as extra conditioning tokens or concatenated with the continuous features. This tests whether the multiplicative role is what matters.

For context at full data, the large vision-language-action models score 96.5% (MemoryVLA) and 96.4% (Discrete Diffusion VLA) average success on LIBERO, and Diffusion Policy reports 95% on Push-T. For subsampled LIBERO at 5 to 25 demonstrations per task, no reported result found on Lacuna, so we will run every baseline ourselves.

Every method gets the same demonstrations, the same frozen encoder checkpoint, the same action head, the same number of gradient steps, and the same number of hyperparameter trials. All are evaluated over three seeds with at least 50 rollouts per task. Beating the strongest baseline means a statistically reliable gain over the best-tuned continuous FiLM at 10 and 25 demonstrations on at least three of the four LIBERO suites, plus a smaller relative drop under visual shift.

Why not just…?

Why not just fine-tune the encoder with low-rank adapters? This is a real threat. VAT shows fine-tuning recovers much of the gap, and adapters limit capacity much as our tables do. What fine-tuning lacks is the per-situation switching. Its single adapted encoder must serve every phase of a task. The demonstration-count sweep settles the question. If adapters match us at 10 demonstrations and under visual shift, the discrete switch adds nothing and we will say so.

Why not just predict FiLM parameters from continuous features, with heavier regularization? This is the most dangerous alternative. A well-regularized continuous generator may do everything the codebook does. The bit-matched information-bottleneck variant tests exactly this. If it ties the discrete version, the honest conclusion is that "bottleneck the modulation" is the finding and quantization per se is incidental. That would still be a useful result, but it is a weaker one, and we would rewrite the claim accordingly. The discrete version has one advantage the continuous one cannot have: small pixel perturbations snap to the same code. Discrete Diffusion VLA's lower vision degradation for discrete actions (20.4% versus 29.0% for continuous diffusion) hints that snapping helps under scene shift. The visual-shift evaluation is where we expect separation, if anywhere.

Why not just condition FiLM on the language instruction alone? This is the $K=1$ endpoint of our sweep, so it is tested for free. It cannot change modulation within an episode. The phase-clock results in PAPL suggest within-episode switching matters, but that was locomotion, not vision. If $K=1$ matches larger $K$ on LIBERO-Long, the learned-phase story is wrong.

Proposed Experiments

The main experiment runs on LIBERO-Spatial, -Object, -Goal and -Long at 5, 10, 25 and 50 demonstrations per task. It sweeps $K \in \{1, 4, 16, 64, 256\}$ with $M \in \{1, 4\}$ and compares against all baselines above. Robustness is measured with LIBERO scene shifts in the manner of Discrete Diffusion VLA's vision-degradation protocol, reported as relative drop from in-distribution success. To check that results are not LIBERO-specific, we repeat the core comparison on Robomimic Square and Transport (proficient-human data subsampled from 200 down to 20 demonstrations), where BC-RNN reaches 84% and 71.3% with full data according to What Matters in Learning from Offline Human Demonstrations, and on Push-T.

The idea succeeds or fails on three checks. The first is an inverted-U in $K$ at low demonstration counts that flattens at 50 demonstrations. The second is the discrete version beating the bit-matched continuous bottleneck under visual shift. The third is code switches aligning with subtask boundaries on LIBERO-Long well above chance. If all three hold, discrete scene codes earn their place as a perception-side analogue of the phase clock. If only the first holds and the bottleneck baseline ties, the contribution narrows to a regularization result for frozen-encoder FiLM. If none holds, frozen encoders plus a good action head are already enough at this scale.