Pixel-Local Cross-Modal Attention for Joint High-Resolution Multimodal Diffusion

Suppose one generative model must produce, or complete, several pixel-aligned views of the same scene at high resolution: color, depth, surface normals, segmentation, edges. The straightforward design turns every modality into its own grid of tokens and lets every token attend to every other. With $M$ modalities and $N$ spatial positions, that costs on the order of $(NM)^2$, so it is quadratic in resolution and quadratic in modality count at once. Missing inputs are usually handled by adding learned parameters for each pattern of which streams are present. The number of such patterns grows as $2^M$. We propose splitting attention along the two axes that cause the blow-up. Tokens from different modalities attend to each other exactly, but only at the same pixel, where cross-modal evidence actually lives: a depth discontinuity sits under a color edge. Long-range context is shared through linear attention, a variant that replaces the pairwise score matrix with a fixed-size running summary of each modality. Summaries from different modalities simply add, so a missing modality is handled by leaving its summary out. No subset-specific parameters are needed.

The payoff is a diffusion model (a generator trained to turn noise into samples by repeated denoising) whose cost grows linearly in $N$ and only as $M^2$ in a small per-pixel term. It accepts any subset of modalities as conditioning and generates the rest. It also avoids the irregular memory gathers that erode the real-world speed of token-pruning and clustering methods. The central empirical question is whether this structure beats two much simpler options: stacking all modalities as channels of one token, and running a strong linear attention over the concatenated sequence. The experiments are designed so that either outcome is informative.

Core Claim

For spatially aligned modalities, most of the useful cross-modal interaction is local in space, and most of the useful global interaction can be shared across modalities. The expensive $M^2$ term can therefore be paid only per pixel, and the $N^2$ term can be removed with linear attention without the quality loss usually attributed to it.

Concretely, let modality $a$ at position $i$ have query $q_{ai}$, and let $\phi$ be a positive feature map. Each observed modality $m$ keeps a summary $S_m=\sum_j \phi(k_{mj})v_{mj}^\top$ and a normalizer $z_m=\sum_j \phi(k_{mj})$. The global path returns

$$o^{\text{glob}}_{ai}=\sum_{m\in\mathcal{O}} g_{am}\,\frac{\phi(q_{ai})^\top S_m}{\phi(q_{ai})^\top z_m},$$

where $\mathcal{O}$ is the set of observed or currently generated modalities and $g_{am}$ are learned scalar gates. The local path is ordinary softmax attention from $q_{ai}$ over the co-located keys $\{k_{mi}: m\in\mathcal{O}\}$. The two paths are summed. Their costs are $O(NMd^2)$ and $O(NM^2d)$ respectively, against $O(N^2M^2d)$ for full joint attention. Here $d$ is the head width.

The design has two deliberate features. First, each modality's summary is normalized separately rather than pooled under one denominator. Our hypothesis is that pooling is exactly what makes linear attention dilute as more streams are summed. Second, missingness and the choice of which modalities to generate come from a single mechanism. Each modality carries its own noise level. Observed streams sit at noise zero and contribute clean summaries; absent streams are dropped from $\mathcal{O}$.

Relation to Prior Work

Three papers motivate the proposal. Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos shows that exhaustively modeling every modality pair with bilinear layers becomes a computational bottleneck. We keep a pairwise term but confine it to a per-pixel set of size $M$. Towards Robust Multimodal Prompting With Missing Modalities shows that one learned prompt per missing-pattern generalizes poorly and grows as $2^M$. Its alternative, one prompt per modality combined by addition, reaches 34.11 F1-Macro versus 20.17 on MM-IMDb under train/test mismatch, but it was tested only with two modalities and only for classification. Our additive composition of linear-attention summaries is the generative, attention-level analogue of that idea. It needs no extra prompt parameters, and we test it at $M\ge 5$. MAESTRO combines sparse attention with modality dropout for robustness to up to 40% missing sensors. It is discriminative and one-dimensional, whereas we target dense two-dimensional generation.

Four papers supply methods. Dynamic Differential Linear Attention shows that linear attention in diffusion transformers blurs detail unless the kernels are sharpened. Its DyDi-LiT-S reaches FID 13.36 on ImageNet 256, against 15.19 for the linear baseline Sana-S and 16.60 for the quadratic DiT-S. We reuse its token differential operator inside each modality's global path. The new question is what happens when several modalities' summaries are combined. Make a Cheap Scaling generates a low-resolution "pivot" and refines it with tiny adapters. We borrow this recipe for the high-resolution stage. Lumina-T2X uses rotary position embeddings and query-key normalization for stable resolution extrapolation. We adopt both. Lumina's attention, however, stays quadratic and it trains a separate model per modality.

Two lines of work constrain the design. An analytic theory of creativity in convolutional diffusion models explains why models built only from local operations tile repeated objects when run at resolutions beyond training. This is why the local path cannot be the only path and a global linear path is required. Xu et al. and Liu et al. show that data movement, not FLOPs, dominates runtime. This argues against grouping schemes that scatter tokens, such as ATD's category attention and AViTS's top-$K$ selection. Our local path reads $M$ contiguous vectors per pixel.

The closest system is CoDi. It runs one latent diffuser per output modality and links them with dense cross-attention plus a shared environment encoder. That design keeps quadratic spatial attention, adds cross-attention channels that grow with modality pairs, and does not model inference-time dropout. We use one shared backbone, no pairwise channels, and dropout as a first-class input. DiffusionDepth is the single-direction special case (depth from RGB) that our model should subsume.

Research Program

We will build a latent diffusion transformer with one lightweight autoencoder per modality, all mapping to a shared token grid. Each block applies the local cross-modal softmax, then the per-modality-normalized linear global path with differential sharpening, then an MLP. The model is trained with flow matching, using an independently sampled noise level per modality. With some probability each modality's noise level is set to zero, which marks it as observed, and with some probability it is removed entirely, which marks it as missing. A single network is therefore trained for joint generation, conditional generation from any subset, and robustness to missing streams.

The analysis has three parts. The first is a dilution study. We measure the entropy of the global-path attention as the number of summed modalities grows, under pooled versus per-modality normalization. This tests directly whether the oversmoothing documented by DyDi-LiT worsens with modality count, as the research file's Gap 3 conjectures. The second is a locality study. We measure how much cross-modal attention mass, in a full-attention reference model trained at low resolution, falls within a small spatial radius of the query pixel. If little of it is local, the core claim fails early and cheaply. The third is a systems study. We fuse the local path into one kernel over contiguous per-pixel tensors and report wall-clock time and peak memory against both FLOP counts and token-pruning methods.

Baselines

The naive reference is channel stacking. All modality latents are concatenated along the channel axis into one token per pixel and fed to a standard DiT at low resolution or the same linear DiT at high resolution. Missing modalities are zero-filled with an indicator channel, and per-modality noise is applied channel-wise. It gets the same data, parameter count, training steps and learning-rate sweep. It is the cheapest possible fusion and it is fully pixel-aligned, so our method must beat it on cross-modal consistency or on unseen missing-patterns to justify its extra structure. Matching it on quality at lower cost does not count.

The strongest existing efficient-attention method is DyDi-LiT's linear attention applied to the concatenated $NM$-token sequence, with the same per-modality noise training (DyDi-LiT). This is the direct test of whether the structural split adds anything over the best linear operator in the file. We will first reproduce its unimodal ImageNet 256 result, where the paper reports FID 13.36 for DyDi-LiT-S, to certify our implementation. It is then trained on the multimodal data under an identical budget. Beating it means lower FID and better cross-modal consistency at equal wall-clock time, with the gap widening as $M$ grows.

The quadratic reference is full softmax joint attention over all $NM$ tokens, trained only at the lowest resolution where it fits. It serves as a quality ceiling, and we would accept being within its noise band. The any-to-any reference is a CoDi-style design with separate per-modality diffusers and pairwise cross-attention, matched in total parameters (CoDi). No reported result is found on Lacuna for CoDi on any aligned dense-modality benchmark, so all numbers will come from our own runs.

For conditional depth we compare against DiffusionDepth, which reports RMSE 0.295 on NYU-Depth-V2 and 1.418 on KITTI. A general any-subset model need not beat a specialist. Coming within about 5% of those numbers while also generating the other modalities would be the bar.

The ablations isolate the key ingredients. We remove the local path, remove the global path, replace per-modality normalization with a pooled denominator, and swap the structural missingness mechanism for per-modality learned missing tokens in the style of modality-specific prompts.

Why not just…?

Why not just stack modalities as channels? This is the real threat. When modalities are perfectly aligned and few, the per-pixel MLP may do all the cross-modal mixing that is needed. Three things plausibly break it. Its width is fixed at design time, so adding a modality means retraining. Zero-filling makes absent streams look like real data, which is exactly the failure that sank missing-pattern prompts on MM-IMDb. And every modality is forced through one shared attention pattern even when geometry and texture need different long-range context. The M-scaling and unseen-subset experiments settle this directly. If stacking matches us throughout, we will report that aligned modalities need no cross-modal attention, which is a useful negative result.

Why not just run DyDi-LiT on the concatenated sequence? Mathematically, linear attention over the concatenation is the sum of per-modality summaries under one pooled denominator. Our method differs in the normalization and in the exact local term. DyDi-LiT's own evidence is that linear attention loses detail when attention spreads out, and pooling $M$ correlated streams is a plausible way to spread it further. The pooled-versus-separate ablation and the dilution-entropy measurement decide whether this matters. The local-path ablation decides whether pixel-precise alignment requires softmax sharpness.

Why not just prune tokens as AViTS does? AViTS keeps quadratic attention and reaches 3.14× speedup on FLUX.1-dev with ImageReward 1.0104 versus 0.9719. The technique is complementary rather than competing. Its gains, however, rely on scattered top-$K$ gathers, which the data-movement studies warn about. It also scores importance from text cross-attention, which does not exist for depth or normals. We will apply AViTS-style selection on top of both our model and the full-attention reference to check that the speedups compose.

Proposed Experiments and Decisive Checks

The main testbed is Taskonomy, which provides many pixel-aligned modalities over roughly 4 million indoor images. We train on RGB, depth, normals, curvature, edges and semantic segmentation, and vary $M$ from 2 to 6. Resolution runs from 256 up to 1024 using the self-cascade pivot recipe. NYU-Depth-V2 and KITTI provide conditional-depth transfer, and ImageNet 256 provides the unimodal sanity check. We know of no reported result on Lacuna for joint multimodal generation on Taskonomy, so every baseline will be run by us.

The metrics are per-modality FID and cross-modal consistency. For consistency, a fixed depth and normal estimator is run on generated RGB and compared by RMSE and angular error against the depth and normals generated alongside it. We also report conditional accuracy from every observed subset, wall-clock time per sample, and peak memory.

Three results are decisive. The first is the scaling curve in $M$. Our cost should grow roughly linearly while concatenated linear attention grows at least linearly with a larger constant and full attention runs out of memory. Quality should hold or improve as $M$ grows. The second is held-out missingness. We train with some modality subsets never used as conditioning and test on exactly those. Structural dropout should degrade gracefully, while learned missing tokens and zero-filled stacking should not. The third is the locality study. If under half of the cross-modal attention mass in the full-attention reference falls within a small spatial neighborhood, we abandon the pixel-local premise before scaling up.