Band-Routed FiLM for Compositional Image Generation

Image generators conditioned on several attributes, such as an object's shape, its color and the background color, tend to break when asked for a combination they never saw in training. Even the best current fix, which gives each attribute its own small conditioning network, still adds every attribute's adjustment into the same internal features. Visual attributes do not all live at the same spatial scale, though. A wall's color is a smooth, slowly varying signal. An object's outline is a sharp edge. A surface pattern is fine-grained detail.

We propose to make each attribute adjust only the frequency bands of the network's internal feature maps where that attribute actually shows up in the data. We would then train the generator with a loss that weighs every frequency band fairly. Under this design, attributes that occupy different scales cannot overwrite each other, and the fine-detail bands receive enough training signal to render unseen combinations sharply instead of as a blurry average.

In two sentences: the existing factored generator gives each attribute its own conditioning head but sums all of them into the same channels. We route each head into a frequency band chosen from the data and pair this with a band-balanced frequency loss. We predict this helps exactly for attribute pairs whose image changes fall in different bands, and does nothing for pairs that share a band. That makes the claim easy to falsify.

Core Claim

The mechanism we build on is FiLM (feature-wise linear modulation): a small network reads a conditioning signal and outputs a per-channel scale $\gamma$ and shift $\beta$ that are applied to a layer's activations. FiLM: Visual Reasoning with a General Conditioning Layer introduced it for visual question answering and showed that its scales act as channel gates. A known limitation is that one scalar per channel covers the whole feature map, so FiLM cannot adjust coarse structure and fine detail in the same channel independently.

The closest existing work is Compositional Flow Matching with Factored Velocity Fields. It trains a flow-matching generator, a model that learns a velocity field transporting noise into images, with a shared UNet and one FiLM head per labelled attribute. The heads' contributions are summed at the UNet bottleneck. That paper supplies our starting architecture and benchmarks. On MPI3D-real with 50% of attribute combinations held out, it reports held-out factor accuracy of about 78% versus about 47% for a single monolithic conditioning network.

Its training loss is a plain pixel-space regression. Its heads all write into the same low-resolution bottleneck channels. And it never asks where in the frequency spectrum its remaining errors on unseen combinations sit.

Our claim has two parts. First, the remaining compositional errors split into two kinds. One is attribute leakage: the wrong color appears because of a memorized co-occurrence. The other is spectral collapse: the unseen combination is rendered as a smooth average with soft edges. Second, the two kinds need two different fixes, and both are required. Band routing, meaning each attribute's FiLM acts only on its assigned frequency bands, reduces leakage between attributes in different bands. A band-normalized frequency loss is needed so that the high-frequency routes get any gradient at all. Without that loss, the low-frequency band dominates the error signal, as Joint Sub-bands Learning with Clique Structures for Wavelet Domain Super-Resolution documented for wavelet sub-bands.

The bridge between frequency and composition is motivated by Seeds of Structure: Patch PCA Reveals Universal Compositional Cues in Diffusion Models. That paper analyzes diffusion models after training and shows that low-frequency components of the noise seed fix scene layout, while high-frequency components change only texture. It uses this for training-free editing. We turn the same observation into a trained architectural constraint.

Research Program

The first component is a data-driven spectral signature for each attribute, so that routing is measured rather than hand-assigned. For attribute $k$, we take pairs of training images that differ only in that attribute. The benchmark datasets enumerate their factors, so such pairs exist. We then compute the share of the difference image's energy that falls in each sub-band $b$ of a two-level Haar wavelet transform:

$$s_{kb} = \frac{\mathbb{E}\,\|W_b(x) - W_b(x'_k)\|^2}{\sum_{b'} \mathbb{E}\,\|W_{b'}(x) - W_{b'}(x'_k)\|^2}.$$

Here $W_b$ extracts sub-band $b$, $x$ is a training image, and $x'_k$ is the same image with only attribute $k$ changed. The wavelet transform splits an image into a coarse approximation band plus horizontal, vertical and diagonal detail bands while keeping spatial position. On Shapes3D we expect wall and floor hue to load almost entirely on the coarse band and shape to load on the detail bands. That expectation is checked rather than assumed, and the signatures themselves are a reportable result.

The second component is the band-routed FiLM layer, inserted at several decoder resolutions instead of only the bottleneck. At each insertion point, the feature map $h$ is split by a one-level Haar transform into bands $h^{(b)}$. Attribute $k$'s head produces band-specific parameters $\gamma_{kb}, \beta_{kb}$. The modulated features are

$$h' = \mathrm{IDWT}_b\Big[\,h^{(b)} + \sum_k a_{kb}\big(\gamma_{kb}(c_k)\odot h^{(b)} + \beta_{kb}(c_k)\big)\Big].$$

In this expression, $c_k$ is the value of attribute $k$, $\mathrm{IDWT}$ is the inverse wavelet transform, and $a_{kb}\in\{0,1\}$ routes attribute $k$ to band $b$ when $s_{kb}$ exceeds a threshold. A soft, learned variant initialized from $s_{kb}$ is studied as an ablation.

Haar transforms on feature maps are a few additions and subtractions. This avoids the heavy recurrent wavelet machinery of SRCliqueNet that the frequency-loss literature flags as a memory cost.

The third component is the training objective. It adds a band-balanced wavelet loss to the standard flow-matching regression, applied to the model's one-step estimate of the clean image: $\hat{x} = x_t + (1-t)\,v_\theta$, where $x_t$ is the noisy image at time $t$ and $v_\theta$ is the predicted velocity. Each band's term is divided by that band's average training energy, in the spirit of Zhong et al.'s "Mode 4" rescaling. Without this normalization, the frequency loss would mostly repeat the pixel loss.

We use wavelets rather than a global Fourier loss because Fourier phase is very sensitive to small spatial shifts. Robot-arm positions in MPI3D vary continuously, which would make that sensitivity a real source of instability.

Baselines

The naive reference point is a monolithic conditional flow-matching UNet that receives all attributes through one conditioning network, as in the Ryoo et al. comparison. It anchors the roughly 47% held-out factor accuracy on MPI3D-real Medium.

The strongest existing method is the factored FiLM model of Compositional Flow Matching with Factored Velocity Fields. We will reimplement it on their Light, Medium and Heavy holdout splits (25%, 50% and 75% of combinations withheld) of Shapes3D and MPI3D-real. We first reproduce their reported MPI3D-real Medium figure of about 78% held-out factor accuracy and their roughly 2.4x FID gain over the monolithic model on Shapes3D Medium. That paper reports only a relative FID figure, not an absolute one, so we will report our own absolute FIDs for every model.

Beating this baseline means a gain in held-out factor accuracy, and a reduction in held-out FID, that exceed seed-to-seed spread on the Medium and Heavy splits. Every model gets the same UNet trunk, the same data, the same number of training steps, three to five seeds, and the same hyperparameter search budget. We will match parameter counts by widening the baseline heads.

Three ablations isolate the key ingredients. The first is the factored model plus our band-normalized wavelet loss with no routing: the loss alone. The second is band routing without the frequency loss: routing alone. The third is band routing with the assignments $a_{kb}$ randomly permuted across attributes. This tests whether gains come from the measured signatures or just from the extra multi-scale structure. A fourth control is spatially-varying FiLM, which predicts a coarse map of scales instead of one scalar per channel, with matched parameters.

Why not just…?

Why not just add a frequency loss to the existing factored model? This is the most serious threat. It is also cheap, and it directly targets spectral collapse. If held-out errors of the factored model sit mainly in the detail bands and are blur rather than leakage, this alternative could match our method. We settle the question two ways. The loss-only ablation is run on equal footing. We also measure leakage directly with per-attribute classifiers on held-out samples, separate from overall sharpness. If the loss alone closes most of the gap in both measures, we will report that routing is unnecessary and the proposal collapses to a simpler, still useful result.

Why not just make FiLM more expressive, with spatial maps or bigger heads? More capacity per head does not stop heads from writing into the same coefficients, which is where we locate leakage. However, a spatially-varying FiLM could in principle learn scale separation on its own. The parameter-matched spatial-FiLM baseline tests this. The paired-band analysis below separates the two explanations: routing should help only on different-band pairs, while extra capacity should help uniformly.

Why not just compose separate generators by adding their energies? Position: Compositional Generative Modeling: A Single Model is Not All You Need argues for summing independent energy factors and sampling with Markov chain Monte Carlo. That approach is principled, but it costs one model per factor and iterative sampling. It also does nothing about the low-frequency bias of each component's regression loss. We include an energy-sum composition of per-attribute conditional flows as a secondary comparison on Shapes3D only. The file contains no result for it on these benchmarks, so its standing is open.

Why not just assign bands by hand, or edit low frequencies after training as Seeds of Structure does? Hand assignment works only when the designer's intuitions are right. Our signatures may show, for instance, that object hue on Shapes3D carries edge energy at object boundaries. Post-hoc low-frequency injection controls layout and object count but has no way to set a named attribute such as wall hue. It also yields blurry structure without texture, per that paper's own limitation.

Proposed Experiments

The first experiment is diagnostic and needs no new model. We train the factored baseline and decompose its held-out-combination error by wavelet band and by attribute. This tells us whether compositional failure is concentrated in particular bands, which nobody in the file has measured.

The main experiment compares all baselines and ablations on Shapes3D and MPI3D-real across the three holdout levels. It reports held-out FID, held-out factor accuracy from attribute classifiers trained on the full grid, per-band reconstruction error, and a leakage score. The leakage score is how often attribute $j$'s classifier is wrong on samples where only attribute $k$ is novel.

The decisive check is a paired-band split. We build holdout splits that withhold combinations of two attributes with disjoint signatures, such as wall hue and shape. We also build splits that withhold combinations of two attributes sharing a signature, such as wall hue and floor hue. The prediction is a clear gain over the factored baseline on disjoint pairs and no gain on same-band pairs. Uniform gains would point to capacity or the loss rather than routing, and we would say so.

A second check repeats Ryoo et al.'s $K \to K+1$ experiment, adding one attribute to a frozen model. We predict that band routing makes it easier to add an attribute whose signature occupies an unused band.

As a stretch test toward natural images, we would use CelebGlow, whose controllable face factors include fine-detail attributes such as hair color. There is no reported generation result found on Lacuna for CelebGlow, so all numbers there will be our own.

We will report an explicit limitation. Attributes that change an object's whole shape, the "melted" or "sliced" states that Troika: Multi-Path Cross-Modal Traction for Compositional Zero-Shot Learning found visually entangled with object identity, should have overlapping signatures. They are therefore outside what routing can help. Likewise, the spatial-relation failures documented in How Clever Is the FiLM Model, and How Clever Can it Be? are not targeted by this work.