Many generators now produce data that is part continuous and part categorical. A 3D molecule has atom coordinates and also atom and bond types, and a document layout has box positions and also element labels. The dominant approach is a joint discrete-continuous diffusion model: a generator that learns to reverse a noising process in which coordinates receive Gaussian noise while categories are randomly resampled or replaced by a [MASK] token, with a single shared network denoising both at once. These models work well, but they rely on hand-tuned tricks that nobody fully explains. The clearest case is MiDi's adaptive schedule, which destroys atom-type information faster than coordinate and bond information (MiDi). They also still produce disconnected fragments on large molecules.
This note proposes a mechanistic explanation and tests it. At high noise, every masked or uniformly resampled categorical token looks the same, so the categorical half of each node's input is identical across all nodes. The network starts from a representation that is already nearly rank one: every node has almost the same features, the same end state that over-smoothing reaches in deep graph networks. The only thing that makes nodes different at that point is the noisy geometry. We claim that sample-quality failures and the need for hand-tuned schedules both follow from how well the denoiser keeps and passes geometric distinctions into the collapsed categorical stream. We further claim this can be measured directly with rank diagnostics taken from the over-smoothing literature, applied per modality, per layer, and per diffusion timestep.
Core Claim
Over-smoothing research asks whether node representations become indistinguishable as depth grows. Inside a joint diffusion denoiser a second axis matters just as much: the noise level. We hypothesize three things. First, the effective rank of the denoiser's categorical-stream hidden states is low at high noise and recovers only as fast as geometry information flows into it. Second, a timestep-resolved rank profile predicts downstream failures such as fragmentation and invalid valency better than depth or parameter count do. Third, a denoiser that keeps separate continuous and categorical streams, joined by cross-attention whose strength is gated by each modality's noise level, can match a hand-tuned adaptive schedule while using a uniform one.
The design has one non-obvious constraint. Suppose categorical tokens attend to geometric tokens with dense cross-attention (queries from one modality, keys and values from the other). Then collapsed queries make the problem worse. If every node's query is the same vector, every node gets the same attention weights and therefore the same output, so the collapse passes straight through. Cross-attention can only break the symmetry if its queries or attention scores depend on node-specific geometry, for example through relative-distance biases or neighborhood restriction. This extends the result of Demystifying Oversmoothing in Attention-Based Graph Neural Networks, which proves that softmax self-attention cannot change graph connectivity and still collapses exponentially. We add a cross-modal failure case that their single-modality analysis does not cover, and we test it directly.
Relation to Prior Work
The joint-diffusion papers supply the setting and the baselines. MiDi diffuses coordinates with Gaussian noise and atom types, charges and bonds with discrete noise, and denoises them with a relaxed equivariant graph network. Its handcrafted schedule is the phenomenon we want to explain. Navigating the Design Space of Equivariant Diffusion-Based Generative Models (EQGAT-diff) shows that discrete categorical diffusion beats continuous relaxation and uses equivariant graph attention. It never inspects hidden-state rank. Equivariant Diffusion for Molecule Generation in 3D (EDM) treats one-hot types as continuous and fragments badly on GEOM-Drugs, which motivates asking what the denoiser actually represents. MDM already splits the network into local and global encoders by a fixed 2Å cutoff. That is a split by distance scale, not by modality, and it has no discrete diffusion, so it is the closest architectural relative but answers a different question. In layouts, DLT uses an absorbing mask for categories, so its masked tokens are identical at high noise. That makes DLT a second, non-graph test of the same mechanism.
The over-smoothing papers supply the measurement tools and constrain the design. Are we measuring oversmoothing in graph neural networks correctly? shows that Dirichlet energy misleads for trained networks, while effective rank (the exponential of the entropy of normalized singular values) tracks accuracy with an average Pearson correlation of 0.91 and flags collapse around layer 10 even with normalization layers present. We reuse this metric, but on generative denoisers and resolved by timestep. Backward Oversmoothing warns that forcing forward features apart can homogenize gradients instead. We therefore avoid adding a rank-maximizing loss. Maximizing rank is also a bad target on its own terms, because random noise has maximal rank and carries no information. We use rank only as a diagnostic, paired with linear probes that check whether the retained dimensions encode the clean categories.
The cross-attention work supplies the gating mechanism. An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention uses a learned gate to stop a large stream from swamping a small one. We couple that gate to the per-modality signal-to-noise ratio, which that work had no reason to do.
Research Program
The first phase is purely diagnostic and needs no new architecture. We train MiDi-style and EQGAT-diff-style denoisers on QM9 and GEOM-Drugs at depths of 4, 8 and 12 layers, each under both the adaptive and a uniform noise schedule. For each trained model we record hidden states at every layer across a grid of timesteps. We split them into the part that updates categorical features and the part that updates coordinates, which is well defined in these architectures because they keep separate node-scalar and coordinate channels. For each part we compute effective rank and numerical rank. For the categorical part we also fit a linear probe that recovers clean atom types. This yields a rank-and-probe profile over layer and timestep for every model. Then, for each generated molecule, we log the rank profile along its own sampling trajectory and test whether low mid-trajectory rank predicts fragmentation or valency failure. That test is per sample, not just per model, which is a much stronger form of evidence.
The second phase builds the two-stream denoiser. Continuous and categorical node features live in separate hidden streams. Each block applies within-stream equivariant message passing and then gated cross-attention in both directions. The attention scores include a learned function of pairwise distances, so that collapsed categorical queries still produce different outputs at different nodes. The gate takes the form
$$h_d \leftarrow h_d + g(\lambda_c^t, \lambda_d^t)\odot \mathrm{CrossAttn}(h_d, h_c, r),$$
where $h_d$ and $h_c$ are the categorical and continuous hidden states, $r$ are pairwise distances, $\lambda_c^t$ and $\lambda_d^t$ are the two modalities' signal-to-noise levels at timestep $t$, and $g$ is a small network with outputs in $[0,1]$. A symmetric update runs in the other direction. The phase-one diagnostics are then rerun on this model.
Baselines
The naive reference point is a MiDi-style shared-stream denoiser trained with a uniform schedule for both modalities. This is what a practitioner gets without MiDi's tuning, and it shows how much the schedule is doing. The strongest existing methods are MiDi with its adaptive schedule and EQGAT-diff's base discrete model. MiDi reports 91.6% molecule stability on GEOM-Drugs; EQGAT-diff cites MiDi at 89.7%. EQGAT-diff reports 91.60% for its base discrete model and above 95% with added chemical features and PubChem3D pretraining. EDM's 5.5% stability, or 40.3% with post-hoc bond optimization, marks the continuous-relaxation floor. Because GEOM-Drugs Revisited shows that fixing the aromatic valency bug lowers stability by 1 to 3% under the refined lookup table, and that kekulized retraining improves validity by about 5%, we will not compare against reported numbers directly. Every model will be retrained from the authors' code under the corrected evaluation, with the same data splits, matched parameter count, matched GPU-hours, and the same hyperparameter search budget for each method. We compare against the non-pretrained EQGAT-diff so that pretraining does not confound the result.
Several ablations isolate the key ingredient. The first gives the shared-stream model both per-modality noise levels as input, with no architecture change. The second keeps the two streams but joins them by concatenation followed by a small network instead of attention. The third uses two streams with cross-attention that has no distance term, which directly tests the collapsed-query prediction. The fourth adds extra residual connections and LayerNorm to the shared-stream model. For layouts, DLT on PubLayNet with category-plus-size conditioning is the baseline, where DLT reports an FID of 11.4 against BLT's 14.2. We will retrain it under matched budget.
Beating the baselines means this: the two-stream model under a uniform schedule matches or exceeds corrected-metric stability and validity of adaptive-schedule MiDi within seed variance on GEOM-Drugs, and also beats every ablation. A win only under the adaptive schedule would be a weaker result, and we will report it as such.
Why not just…?
Why not just keep tuning the noise schedule, as MiDi does? This works. The question is whether it is a workaround for a representational bottleneck. If the uniform-schedule shared model shows low categorical-stream rank and poor probe accuracy at mid timesteps, and the adaptive schedule mainly shifts where that bottleneck falls, the schedule is compensating for the bottleneck rather than removing it. If the two-stream model is schedule-insensitive, that shows the architectural fix generalizes where hand-tuning does not. If adaptive-schedule MiDi still wins under matched budget, the architectural claim fails. We would then report the diagnostic alone as the contribution.
Why not just feed both noise levels to the shared network? This is the most serious threat. It is cheap, and a shared network conditioned on both noise levels might learn the same gating internally. The first ablation tests exactly this. If it matches the two-stream model on stability and on the rank profile, we will conclude that the missing ingredient was information about the noise levels, not separate streams or cross-attention, and the proposal narrows to the diagnostic plus that recommendation.
Why not just add residuals and normalization, or stay shallow? Are we measuring oversmoothing in graph neural networks correctly? found that effective rank still collapses by layer 10 with BatchNorm and LayerNorm. Those results come from node classification, though, not denoisers. The residual-plus-LayerNorm ablation and the depth sweep test whether they carry over. Staying shallow limits how far information can travel in molecules of around 44 atoms, but that is an assumption, and the depth sweep checks it.
Why not just use dense cross-attention between modalities? Our argument predicts that collapsed categorical queries make dense cross-attention hand the same output to every node. The distance-free ablation measures this directly as categorical-stream rank at high noise. If that ablation does not collapse, the mechanism is wrong and the design rationale must be revised.
Decisive Checks
On QM9 and GEOM-Drugs, with evaluation corrected as in GEOM-Drugs Revisited, we report molecule stability, validity, fragment rate, and bond-length and bond-angle deviations. For every model we also report rank-and-probe profiles over layer and timestep. Three results would settle the hypothesis. First, the per-sample relationship between mid-trajectory categorical-stream effective rank and fragmentation should be clearly monotone on GEOM-Drugs. It should be weaker or absent on QM9, where molecules are small enough that local message passing reaches every atom. Second, the distance-free cross-attention ablation should show near-rank-one categorical states at high noise, while the distance-aware version should not. Third, the two-stream uniform-schedule model should close the gap to adaptive-schedule MiDi, while the noise-level-conditioning ablation does not.
As a transfer test, we repeat the diagnostics on DLT for PubLayNet and RICO, measuring FID, overlap and alignment. Masked layout tokens are identical at high noise, so the same rank-recovery pattern should appear in a network with no graph at all. If it does not, the mechanism is specific to message passing and we will say so.
References
- MiDi: Mixed Graph and 3D Denoising Diffusion for Molecule Generation
- Demystifying Oversmoothing in Attention-Based Graph Neural Networks
- Navigating the Design Space of Equivariant Diffusion-Based Generative Models for De Novo 3D Molecule Generation
- Equivariant Diffusion for Molecule Generation in 3D
- MDM: Molecular Diffusion Model for 3D Molecule Generation
- DLT: Conditioned layout generation with Joint Discrete-Continuous Diffusion Layout Transformer
- Are we measuring oversmoothing in graph neural networks correctly?
- Backward Oversmoothing: why is it hard to train deep Graph Neural Networks?
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- GEOM-Drugs Revisited: Toward More Chemically Accurate Benchmarks for 3D Molecule Generation