Coarse-Guided Local Channel Gating for Next-Scale Image Generators

Coarse-to-fine image generators draw an image as a small grid of tokens and then fill in progressively larger grids, each conditioned on the ones before. They adjust their internal feature channels in a way that is the same everywhere in the image. Every token at a given scale receives the same per-channel scale and shift, computed from the class label or from a pool over the whole image. Video models that work this way have a documented failure mode. When features are averaged over the whole frame before deciding what matters, small foreground objects get drowned out by large background regions. We propose to test whether next-scale generators have the same problem, and to fix it cheaply. The fix makes the channel gate vary by location, computing it from the coarser scale the generator has already produced. It pools over a small neighborhood instead of the whole image.

In two sentences: the channel-attention blocks used to recalibrate features, and the class-conditioning layers inside next-scale transformers, both assume one channel weighting fits every location. We replace that with a gate whose locality is a single tunable radius, predicted from the previous scale. We then measure whether image quality, fine-detail fidelity and small-instance accuracy improve as the radius shrinks. A global gate is the special case in which the radius covers the whole grid. That makes the central claim directly falsifiable: if the global setting does as well as the local ones, the claim is wrong.

Core Claim

The hierarchical generator here is Visual Autoregressive modeling (VAR), from Visual Autoregressive Modeling. VAR predicts whole token maps from $1\times1$ up to $16\times16$, conditioning each on previous scales, and reports FID 1.73 on ImageNet $256\times256$. Its follow-ups reshape how tokens attend across scales. M-VAR splits within-scale attention from a linear-time state-space path across scales. MVAR conditions only on the immediately preceding scale, through a fixed $k\times k$ window, removing the key-value cache. FVAR replaces digital downsampling with optical-style blur to avoid aliasing. None of these changes how channels are weighted, which the research file marks as absent across the whole next-scale literature.

On the attention side, CBAM supplies the method we build on and also the assumption we challenge. It computes a channel gate from global average and max pooling over the full grid, then a spatial gate from a $7\times7$ convolution. It shows that channel-then-spatial ordering beats the alternatives and that both parts help, cutting ResNet-50 top-1 error from 24.56% to 22.66% against 23.14% for channel-only gating.

The spatial-granularity literature motivates doubting global pooling. Action recognition with spatial-temporal discriminative filter banks shows global aggregation washes out subtle distinctions between actions. Surgical Skill Assessment via Video Semantic Aggregation shows it dilutes small-tool signals against background tissue. Diversifying Spatial-Temporal Perception shows it lets static backgrounds dominate. MV-Former recovers fine-grained performance by pooling into a few learned entity slots instead of one vector.

Our claim is that a generator refining a small object at scale $s$ needs different channels emphasized there than in the surrounding background. It also has, for free, a spatial map telling it where that object is: the committed scale $s-1$. We claim a spatially varying channel gate read from that map improves fine-detail fidelity at negligible cost. Because the gate reads only the previous scale, it keeps MVAR's cache-free, Markovian inference.

Mechanism

Let $h_s \in \mathbb{R}^{H_s\times W_s\times C}$ be the features at scale $s$, with height $H_s$, width $W_s$ and $C$ channels. Let $u_{s-1}$ be the previous scale's output features, upsampled to $H_s\times W_s$. At each block we compute

$$ g_s = \sigma\big(W_2\,\phi(W_1\,[\mathrm{Avg}_r(u_{s-1});\ \mathrm{Max}_r(u_{s-1})])\big), \qquad h_s \leftarrow h_s \odot (1 + g_s), $$

where $\mathrm{Avg}_r$ and $\mathrm{Max}_r$ are average and max pooling over an $r\times r$ neighborhood around each location. Here $W_1$ and $W_2$ form a small shared per-location layer, $\phi$ is a nonlinearity, $\sigma$ is the sigmoid, and $g_s \in \mathbb{R}^{H_s\times W_s\times C}$ is the gate.

The gate is applied after the existing class-conditioned normalization, which is kept as is. Setting $r$ to the full grid reproduces CBAM's channel gate exactly. Small $r$ gives a local gate. The pooling pair follows CBAM's argument that max pooling keeps sharp local cues. A CBAM-style spatial gate follows the channel gate, preserving CBAM's ordering. The added parameter count is on the order of $C^2/8$ per block, which is small next to one attention layer.

Research Program

The work has three stages.

The first stage is diagnostic and establishes whether the pathology exists. On pretrained VAR-d16 and MVAR-d16, we measure how per-channel feature statistics vary across locations at each scale, and whether the uniform class modulation accounts for most of the channel variance. We then locate where generation errors concentrate at the finest scales. To do this we score local crops of generated images separately from whole images, split into high-detail and low-detail crops by edge density. The prediction borrowed from the video literature is that errors concentrate in small, high-detail regions.

The second stage trains VAR-d16 and MVAR-d16 from scratch on ImageNet $256\times256$ with the gate inserted, sweeping $r \in \{1, 3, 5, \text{full}\}$. Everything else is identical to the published recipes.

The third stage tests generality where small structures are explicitly labeled. We insert the same gate into the hierarchical label-map generator of Seg-VAR, which decodes segmentation coarse-to-fine, and evaluate on COCO panoptic and instance segmentation. There, "fine granularity" means small instances with ground-truth masks.

Baselines

The naive reference is the unmodified generator. MVAR and FVAR both report FID 3.55 for vanilla VAR-d16 on ImageNet $256\times256$. We retrain it ourselves under our exact schedule so that every comparison uses the same data, epochs and evaluation code.

The strongest method compatible with our module at this scale is MVAR-d16, which reports FID 3.09 in MVAR. It already exploits spatial locality across scales, so it is the competitor most likely to have absorbed any benefit locality offers. We add the gate on top of MVAR and compare against MVAR retrained under our schedule. FVAR-d16, which reports FID 2.89 in FVAR, is the strongest d16 number in the file. It changes the pyramid rather than the block, so it is orthogonal. We report it as a reference and, compute permitting, test the gate on top of it. At d30, MVAR-d30 reports FID 1.33. We will only claim improvement at that scale if we run it.

The key ablation is the same module with $r$ set to the full grid, which is a CBAM gate inserted into VAR. This isolates locality from simply having extra gating. A second ablation feeds the gate from the current scale's own features rather than the previous scale's. This isolates the use of the coarse map as the source of spatial guidance.

All variants receive an equal tuning budget of three learning-rate settings. For each comparison, we give the unmodified model extra width so that parameters and training compute match the gated model.

Beating a baseline means a lower FID than that baseline, with the gap exceeding the seed-to-seed spread over three seeds. It also requires that local-crop FID on high-detail crops improves by more than whole-image FID does. The second condition is the signature of reduced spatial granularity loss rather than a generic gain. On COCO, Seg-VAR reports 59.7 PQ panoptic and 52.7 AP instance. Beating it means gains concentrated in small-object AP.

Why not just…?

Why not just insert CBAM or squeeze-and-excitation blocks as they are? This is the most serious threat. It is exactly the $r=\text{full}$ arm of our sweep, run at matched compute. If full-grid gating matches small-$r$ gating, then locality is not the ingredient. In that case we would report the result as "channel attention helps next-scale generators" and drop the granularity claim. The evidence for expecting locality to matter comes from the video papers cited above, not from any generation result. The file contains no measurement of CBAM inside VAR, so the sweep settles the question empirically.

Why not just widen MVAR's window or use dense cross-scale attention, since attention can already route location-specific information? MVAR reports that dense VAR-d30 does worse than its $k\times k$ restriction, with FID 2.65 against 1.33. Simply adding attention reach is therefore not evidently helpful. Attention mixes token values but applies the same feed-forward channel weighting at every location, and that is what our gate changes. To test the point directly, we run MVAR with a larger window at compute matched to our gated model. If the larger window closes the gap, our gate is redundant with attention.

Why not just scale the model or the tokenizer? Width-matched baselines control for parameters. The published FVAR aliasing analysis argues that pure capacity cannot fix pyramid-induced detail loss, but that is a separate cause. We therefore also test the gate on FVAR. Gains that persist on top of the anti-aliased pyramid indicate the gate addresses something aliasing does not.

Decisive Checks

The main experiment is the locality curve on ImageNet $256\times256$ at d16. It plots whole-image FID, high-detail local-crop FID, precision and recall against $r$, for both VAR and MVAR. The proposal succeeds if small $r$ beats full-grid gating and the width-matched baselines on crop-level metrics, with no loss in whole-image FID. We found no reported result on Lacuna for local-crop FID of any next-scale model, so all crop numbers will be our own.

We also verify that the gated MVAR stays cache-free and report its sampling throughput. On COCO with Seg-VAR, the decisive quantity is small-object AP relative to large-object AP. Seg-VAR reports 3.2 FPS, and we report speed alongside accuracy.

Finally, we visualize the gates. If they carve images into object-aligned regions, as MV-Former's slots do, that supports the mechanism. If they come out spatially flat even when $r$ is small, the model is not using locality, and that also counts as evidence against the hypothesis.