Synergy-Scheduled Guidance for Discrete Diffusion in Combinatorial Design

Generative design optimization replaces point search with sampling: train a diffusion model — a generator that learns by gradually denoising random noise into samples — on a corpus of valid designs, then nudge the denoising trajectory toward desired properties, a nudge called guidance. When the desired properties arrive through different channels — a continuous property predictor pulling the molecule toward a target HOMO-LUMO gap, and a discrete symbolic constraint demanding a particular scaffold or symmetry — current methods blend the channels with fixed, hand-tuned weights. The result is the same failure mode documented across multimodal learning: the easily-optimized channel dominates and the other is ignored, or both channels push the same tokens and waste the guidance budget. Meanwhile, the theory of discrete guidance tells us when to push (late, not early) but says nothing about which channel to push for which sample.

This proposal connects the two directions by making cross-modal synergy an active controller for discrete property guidance. The core mechanism: at every denoising step, cheaply estimate whether the guidance channels are redundant (saying the same thing about the evolving design) or synergistic (each constraining a different aspect, so that only their joint use pins down a solution — the "geometric triangulation" effect), and set per-channel guidance weights accordingly. This turns two heuristic practices — the fixed guidance schedules of discrete diffusion and the fixed modality dropout of complementarity-aware fusion — into a single measurable, sample-adaptive rule.

Core Claim

Discrete diffusion guidance weights should be a function $w_i(t, x_t)$ of time, channel, and the current partially-denoised sample, not a constant. Two results motivate this. First, Theory-Informed Improvements to Classifier-Free Guidance for Discrete Diffusion Models proves that naive guidance in discrete diffusion accelerates token commitment early, when context is missing, and that late-heavy "Ramp-Up" schedules fix this — but their schedule is a time-only heuristic, uniform across samples and conditions. Second, Efficient Quantification of Multimodal Interaction at Sample Level shows via LSMI that interaction profiles are sharply sample-dependent (some samples are redundancy-dominated, others synergy-dependent) and that out-of-distribution generalization tracks the synergistic atom specifically. Nobody has combined these: guidance schedules today treat every sample and every conditioning channel as interchangeable.

We claim that scheduling discrete guidance by an amortized, sample-level redundancy–synergy decomposition will beat the best fixed and time-only schedules at matched compute, and that a companion redundancy penalty will curb the surrogate exploitation that plagues offline design optimization. The supporting diagnostic evidence already exists but is passive: Diffusion PID: Interpreting Diffusion via Partial Information Decomposition shows that when a diffusion model fails to separate distinct prompt conditions (concept merging), the intermediate representation shows abnormally high redundancy and near-zero synergy — yet it only measures this after the fact, at roughly 7 seconds per image on an NVIDIA A6000, which is far too slow to steer a multi-step sampler. We propose to make this measurement fast and to act on it.

Research Program

The system has three components, all built on an existing discrete diffusion backbone rather than a new architecture.

First, a multimodally-conditioned discrete denoiser. We use DiGress-style discrete Markov transitions over categorical node and edge attributes (DiGress: Discrete Denoising diffusion for graph generation), which preserve graph sparsity and permit auxiliary structural descriptors (cycle counts, Laplacian spectra) at every intermediate step, combined with the column-normalized transition rates of the theory-informed CFG work, which prevent premature unmasking. Conditioning arrives on two deliberately distinct channels: a continuous property channel (scalar target values, via external regressor gradients, exactly as DiGress steers toward dipole moment and HOMO-LUMO gap) and a discrete symbolic constraint channel (scaffolds, symmetry and complexity hints, operator or substructure constraints, in the spirit of the symbolic hints in Controllable Neural Symbolic Regression). This pairing is chosen to reproduce, inside a diffusion sampler, the triangulation that NSRwH achieved autoregressively, and to expose the modality-collapse risk between a dense continuous signal and a sparse discrete one — the same imbalance that Visual-Geometry Diffusion Policy had to fight between RGB and point clouds.

Second, an amortized synergy estimator. Off-line, we run LSMI-style pointwise decomposition (with KNIFE conditional-entropy estimation) on training samples to produce ground-truth redundancy $\hat{R}$, uniqueness $\hat{U}_1, \hat{U}_2$, and synergy $\hat{S}$ atoms between the two conditioning channels with respect to the realized design; LSMI makes this feasible where batch PID would not — LSMI reports roughly 11 minutes on UCF-101 versus over 13 hours for the prior estimator. We then distill these estimates into a small head that predicts the atoms from the cached intermediate features (graph descriptors, masked-token statistics, current channel predictions) in one forward pass, sidestepping DiffusionPID's per-step gradient-access latency.

Third, the controller. Each channel's guidance weight at reverse step $t$ is set to

$$w_i(t, x_t) \;=\; w_{\max}\,\rho(t)\,\frac{\hat{U}_i(t,x_t) + \tfrac{1}{2}\hat{S}(t,x_t)}{\hat{R}(t,x_t) + \hat{U}_1(t,x_t) + \hat{U}_2(t,x_t) + \hat{S}(t,x_t) + \epsilon},$$

where $\rho(t) = t/T$ is the late-heavy time ramp (kept from the theory-informed result, which we do not dispute), $\hat{U}_i$ is the amortized estimate of information channel $i$ alone contributes about the current sample $x_t$, $\hat{S}$ the joint-only contribution, $\hat{R}$ the duplicated contribution, and $\epsilon$ a stabilizer. A channel that merely duplicates the other (high $\hat{R}$) is throttled; a channel carrying unique or synergistic information keeps full late-stage strength. For the offline design-optimization setting we additionally test a redundancy penalty — down-weighting guidance steps where the property surrogate's signal is almost entirely redundant with the denoiser's own pull — motivated by the finding in Design-Bench that optimizers exploit correlated surrogate features and produce "adversarial designs" that score high on paper and fail in validation, and by Rosas et al.'s observation in Synergistic information supports modality integration and flexible learning in neural networks solving multiple tasks that redundant and synergistic representations have genuinely different functional roles.

Proposed Experiments

Three settings, ordered from mechanism validation to application. On QM9 with DiGress as the backbone, we condition on paired targets (HOMO-LUMO gap plus dipole, plus symbolic constraints such as ring counts and scaffold presence). The mechanistic check is correlational: does per-sample property hit-rate and validity track the estimated synergy atom, as LSMI's OOD findings predict? The intervention check is whether synergy-scheduled weights beat the best fixed-weight and best ramp-up-only schedules at matched validity. The relevant reference numbers from the file: DiGress reaches 99% validity in 1 hour of training where its continuous counterpart needed 7.2 hours, and its property conditioning cut mean absolute error on targeted properties like the HOMO-LUMO gap by nearly 50% relative to unconditional generation; on GuacaMol it reached a Fréchet ChemNet Distance of 68.0. Our target is at least a further 15% reduction in MAE on double-constraint targets at matched validity, plus GuacaMol distribution-learning checks to confirm no fidelity cost. On AI Feynman / FSReD symbolic regression, we build the discrete-diffusion remake of NSRwH: an equation-token diffusion conditioned on a numerical $(x,y)$ encoder and a symbolic-hint encoder. This isolates the synergy hypothesis cleanly, because NSRwH already observed super-additive gains from combining hints without formalizing them. Reference points: AI Feynman solved all 100 Feynman-lecture equations and 90% of the 20 bonus equations (Eureqa: 71% and 15%), and was robust up to $10^{-4}$ relative noise; no reported result found on Lacuna for NSRwH's exact per-benchmark recovery percentages, so we will re-run an autoregressive baseline ourselves under matched data and compute. On the discrete Design-Bench tasks (TF Bind 8, TF Bind 10, ChEMBL, NAS), we evaluate whether the redundancy penalty reduces the gap between proxy-oracle score and held-out true-oracle score — the adversarial-design signature. No reported result found on Lacuna for exact per-task numerical values on Design-Bench; the file only records qualitatively that CMA-ES and normalized gradient ascent were competitive in 4 of 8 tasks and that Conservative Objective Models won on the high-dimensional Hopper Controller task, so all comparisons will be re-instrumented by us under a fixed evaluation protocol.

Baselines

The do-nothing reference is unconditional discrete diffusion with rejection sampling against the target properties: sample freely, keep designs that happen to meet the targets. This is the floor that DiGress's own property conditioning already improved on by nearly 50% in MAE, and it calibrates how much of any gain comes from conditioning at all. The strongest existing competitor is DiGress with its fixed-strength property-regressor guidance on the same backbone and data — same discrete transitions, same auxiliary features, same training budget — because it is the closest published method for steering discrete graph diffusion with property predictors. We will run it with a hyperparameter search over its fixed guidance strength equal in GPU-hours to the budget we spend on our controller, so the comparison is between scheduling rules, not tuning effort. The strongest schedule-only competitor is the Ramp-Up discrete CFG of the theory-informed paper, run at matched peak weight; beating it at matched budget is the decisive test of whether sample-level information adds anything beyond time-only scheduling. For multi-objective discrete tasks we compare against Pareto-Conditioned Diffusion, which achieved the best average rank across five offline multi-objective task categories and up to 20x faster sampling than ParetoFlow, while noting it treats discrete structures through continuous logit relaxations (no numeric hypervolume values appear in the file). The key ablation is our own architecture with the synergy head disabled and weights frozen at the time-average of the scheduled values: any remaining gain then comes from the time ramp or the backbone, not from the synergy signal. A second ablation freezes $\rho(t)$ to a constant, isolating the per-sample channel allocation from the time schedule. What counts as winning: on QM9, at least 15% lower MAE than the best fixed-weight DiGress at 99% validity; on AI Feynman, exact-recovery rate above our matched autoregressive baseline under identical hint budgets; on Design-Bench, higher true-oracle 100th-percentile performance without validity collapse.

Why not just…?

Why not just relax the discrete tokens to continuous logits and run standard Gaussian diffusion? This is the simplest implementation route, and it is exactly what PCD does for discrete tasks — but its own authors report that the continuous relaxation is sub-optimal on discrete sequence and architecture tasks and identify native discrete diffusion as required future work. DiGress gives the concrete mechanism of failure: Gaussian noise on adjacency matrices turns sparse graphs into a dense float "soup," destroying the sparsity and cycle counts that intermediate structural hints depend on, and it needed 7.2 hours of training to reach the 99% validity that discrete transitions reached in 1 hour. Since our contribution is a guidance controller, not a denoiser, adopting the relaxed backbone would corrupt the very intermediate structural signals the synergy head reads. We will still run a logit-relaxed variant as an ablation, since it is a real threat on sequence-only tasks like TF Bind where graph structure is absent.

Why not just use an autoregressive multimodal Transformer with a concatenated prompt? NSRwH shows this works well when hints are present: distinct symbolic hints combined super-additively on AI Feynman benchmarks. It is the most serious alternative for the symbolic-regression experiment, and we treat it as a head-to-head baseline rather than an argument. What it lacks is any mechanism to re-weight channels mid-generation or to pull exploratory candidates back toward the data manifold; its authors concede a "No Knowledge Floor" where missing or noisy hints collapse performance to black-box baseline level, whereas a diffusion sampler retains the score's validity envelope and, in our design, redistributes weight across channels when one is uninformative. If the autoregressive baseline matches our recovery rate at equal compute and hint budget, the honest conclusion is that synergy scheduling matters only for iterative refinement, and we would report that.

Why not just use fixed guidance weights or a fixed heuristic schedule, possibly with fixed modality dropout? This is the strongest simple alternative and our primary baseline, not something to argue around. The evidence against purely fixed fusion is indirect but consistent: VGDP's fixed dropout of $p=0.20$ prevented collapse in robotics, but Rosas et al. show uniform dropout actively erodes the synergistic representations that multi-source integration needs, and VGDP's own dropout value is untuned per task. LSMI shows interaction profiles vary sample-by-sample, which fixed schedules cannot express. Still, if ramp-up with a well-tuned constant channel ratio matches our controller on QM9 and AI Feynman at matched tuning budget, the proposal's central claim is falsified, and the experiments are designed so that this outcome is visible rather than absorbed by noise.

Why not just scale up a single-condition model, or concatenate the two condition embeddings and train end-to-end? The file contains direct counter-evidence: LSMI's authors state that increasing the complexity of a single-modality model does not replicate the benefits of adding a modality, and Rosas et al. show synergistic spikes in neural networks are driven by the requirement to integrate distinct non-redundant sources, not by parameter depth. Naive concatenation also risks the modality collapse VGDP documented, where policies relying on one stream dropped below 20% success under viewpoint shift while complementarity-aware fusion held 64%. Our setting has the same imbalance (a dense continuous property signal versus a sparse discrete constraint signal), which is precisely why we expect concatenation to underperform here too — but we will run the concatenated-embedding variant as an ablation to confirm rather than assume it.

Why not just use gradient ascent on a surrogate, or CMA-ES? On some offline design tasks this genuinely suffices — Design-Bench's own authors found CMA-ES and normalized gradient ascent competitive in 4 of 8 tasks, dominant on TF Bind and Superconductor. But the same study shows gradient ascent fails on high-dimensional control surfaces through adversarial surrogate exploitation, and PCD found these classical baselines catastrophic on other task categories while achieving the best average rank. Our Design-Bench experiments include both as cheap baselines; on TF Bind we may well lose to them and will say so, but the redundancy-penalty mechanism is aimed at the adversarial-design failure they share, which we measure with a held-out true oracle.

Decisive Checks

The proposal stands or falls on three numbers. First, on QM9 double-constraint targeting, the synergy-scheduled controller must beat the best fixed-weight and ramp-up schedules by a margin outside seed variance at matched validity — otherwise sample-level information adds nothing over time-only scheduling. Second, on AI Feynman, the diffusion-based multimodal sampler with synergy scheduling must at least match the matched autoregressive baseline, and the per-sample synergy estimates must correlate with equation recovery strongly enough to be predictive, not merely descriptive. Third, on Design-Bench discrete tasks, the redundancy penalty must shrink the proxy-to-true-oracle gap relative to unpenalized guidance, since that gap is the measurable signature of the adversarial designs that motivate the whole approach. A negative result on any of these is informative: it would bound how far the redundancy–synergy decomposition, which has so far explained multimodal behavior post hoc, can be pushed into an active control signal for discrete generation.