When a regression model is asked to predict an output that has many valid answers, it does something quietly destructive: it predicts the average of those answers. In speech synthesis this is called statistical over-smoothing, and it ruins voices — formants blur, pitch flattens, everything sounds muffled and robotic. The exact same failure occurs in expensive black-box optimization, where an optimizer must find a design that maximizes an unknown, costly-to-evaluate objective using only dozens or a few hundred queries. There, surrogate models that average over neighboring candidates smooth away sharp, needle-like optima and misdirect the search. Two research communities have independently diagnosed the same pathology — conditional-mean collapse — and even developed similar cures, without noticing each other.
This proposal imports the cure from audio into the optimizer. The bridge is the Trajectory Intersection Phenomenon discovered in singing-voice synthesis: DiffSinger showed that the blurry, over-smoothed output of a fast regression model is not garbage but a structural skeleton — if you add the right amount of noise to it, it lands on the trajectory that a diffusion model (a generator that learns by gradually denoising random noise into samples) would traverse toward the true data. Reverse diffusion can then start partway along, at step $k$ instead of the maximum step $T$, recovering the sharp details the regression averaged away at a fraction of the cost. Meanwhile, Diffusion Models for Black-Box Optimization (DDOM) showed in the offline setting that conditional diffusion, rather than deterministic regression, is the right way to map a desired objective score back to designs, because that mapping is one-to-many and regression collapses it to invalid averages. Nobody has combined the two: every existing diffusion-based optimizer starts from pure noise, and none operates in the active, closed-loop setting where queries are chosen dynamically.
Core Claim
The claim is that in active black-box optimization, the over-smoothed prediction of a cheap regressor — traditionally treated as a defect to be corrected — is in fact a free warm start for diffusion-based candidate generation, and that exploiting it yields better designs per query and per unit of compute than both standard Gaussian-process Bayesian optimization and a directly online-adapted DDOM. Concretely, we build an optimizer that maintains a conditional diffusion model $p(\mathbf{x} \mid y)$ over designs, trained on accumulated (design, score) observations with rank-based loss reweighting as in Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining, and generates each round's candidates not from pure noise but from a noised skeleton:
$$\mathbf{x}_k = \sqrt{\bar\alpha_k}\,\hat{\mathbf{x}}(y^*) + \sqrt{1-\bar\alpha_k}\,\boldsymbol\epsilon, \qquad \boldsymbol\epsilon \sim \mathcal{N}(0, I),$$
where $\hat{\mathbf{x}}(y^*)$ is the over-smoothed output of an auxiliary regressor trained to predict designs from scores, $y^*$ is an extrapolated target score above the best seen so far (steered at inference by classifier-free guidance, following DDOM), $\bar\alpha_k$ is the standard diffusion noise schedule, and $\boldsymbol\epsilon$ is Gaussian noise. Reverse diffusion then runs only from step $k$ to step $0$, restoring the sharp, multimodal structure that the regressor averaged away. The smaller $k$ is, the cheaper the sample; the right $k$ is the one that intersects the true high-scoring trajectory, and we hypothesize it is small exactly because the regressor skeleton is already "on-manifold but blurry" — the same geometry DiffSinger exploited to cut inference cost by 45.1% (real-time factor from 0.348 to 0.191) on the PopCS singing corpus.
The method, which we call shallow-diffusion Bayesian optimization, has two further ingredients. First, acquisition: candidate batches of $J$ shallow samples are scored by a small Gaussian process fitted directly on design representations (not on any latent space, heeding the geometric failure modes Return of the Latent Space COWBOYS documented), and ensemble disagreement across the $J$ samples — high when the diffusion model itself is uncertain which mode of the one-to-many mapping to produce — serves as an epistemic exploration signal alongside the GP variance. Second, an evidential extension: following the Normal-Inverse-Gamma construction of Bayesian Speech synthesizers Can Learn from Multiple Teachers (BELLE), the skeleton regressor can emit per-coordinate variances, so that coordinates where the regressor is confident get a late, cheap diffusion start while uncertain coordinates get an earlier, more diffused one — a per-dimension choice of $k$.
Research Program
The work has three phases. Phase one builds the core loop: the conditional diffusion model with rank-reweighted training, the skeleton regressor, the classifier-free-guidance extrapolation knob for setting $y^*$, the GP scorer on raw designs, and the batch-disagreement acquisition. The central engineering question is the choice of $k$: we will treat it as adaptive, selecting $k$ per iteration by validating on held-out observed pairs (pick the $k$ whose reverse-diffusion samples best reconstruct known high scorers), and we will characterize how the optimal $k$ moves as observations accumulate and the regressor sharpens. Phase two develops the evidential, per-coordinate schedule, which requires the skeleton regressor to output distribution parameters rather than points; here BELLE's Student-$t$ machinery is reused essentially unchanged, transplanted from mel-spectrogram frames to design coordinates. Phase three adds an analysis component: a scaling study, in the spirit of Beyond Oversmoothing: Evaluating DDPM and MSE for Scalable Speech Synthesis in ASR, asking whether the shallow mechanism changes the asymptotic story or only the constant factors — Minixhofer et al. showed that adding data to an MSE-trained generator merely makes it "more precisely average" (plateau at a Word Error Rate Ratio of roughly 2.4–2.5, versus 1.46 for a diffusion model at 5,000 hours of LibriHeavy training data), and we expect an analogous ceiling for the skeleton regressor that the diffusion refinement lifts.
A secondary demonstration grounds the audio direction as an application rather than a mechanism: subjective listening evaluation is itself an expensive black-box objective, since each human rating panel costs real money and time. We will optimize a small set of discrete and continuous generation settings of a TTS pipeline (sampling temperature, guidance scale, vocoder variant) against small AB-preference panels, with mean opinion score as the target metric — the same metric under which Towards Robust FastSpeech 2 by Modelling Residual Multimodality reported lifting LibriTTS quality from 3.23 to 3.41 with its TVC-GMM. This is a genuinely black-box, expensive, and multimodal objective where gradient-based approaches like Optimising Hearing Aid Fittings for Speech in Noise with a Differentiable Hearing Loss Model are structurally unavailable.
Proposed Experiments
The primary testbeds are drawn from the file. First, active-query versions of Design-Bench tasks, where the offline dataset is replaced by oracle access with a 500-query budget: the Superconductor critical-temperature task (on which DDOM beat the next best offline baseline by 11%), TFBind8 DNA affinity, and ChEMBL molecule design. The metric is the best normalized true score found within budget, plus wall-clock per iteration. Second, the Needle-in-a-Haystack testbed of Fast Bayesian Optimization of Needle-in-a-Haystack Problems using Zooming Memory-Based Initialization: the 144-permutation Ackley suite (2D to 10D, needle widths 0.001% to 100% of volume), the 6D auxetic-materials and thermoelectric figure-of-merit tasks over 146,000 materials, and the 11D wildfire task, with budgets of 100 to 900 experiments. These are the landscapes where surrogate smoothing is fatal, and where the reported bar is concrete: standard leading BO methods (TuRBO, HEBO, MiP-EGO) failed to find the 6D Ackley global minimum while ZoMBI succeeded within 900 experiments. Third, for the audio demonstration, AB-preference panels on LJSpeech-based synthesis with roughly 30 queries.
The decisive checks are three. The first isolates the shallow start: the identical system run with diffusion started at $T$ from pure noise (an online DDOM) versus at adaptive $k$ from the skeleton, at matched query budget, matched diffusion architecture, and matched retraining schedule; if the full-start version matches the shallow one on best-found score at equal wall-clock, the central hypothesis is falsified. The second isolates the skeleton's origin: replace the over-smoothed regressor prediction with the best design seen so far as the warm start; the hypothesis predicts the regressor skeleton wins because it averages multiple good modes and is conditioned on the extrapolated target, whereas the best-seen point anchors the search to a single mode. The third isolates the reweighting: train the diffusion model on uniform weights, testing Tripp et al.'s claim that static generative capacity is misallocated without rank weighting — while noting that Moss et al.'s counter-evidence (a frozen VAE suffices once the surrogate is fitted in the right space) means this ablation could plausibly go either way, which is precisely why we run it.
Baselines
The naive reference point is an initial random design plus random search for the remainder of the budget: on the active Design-Bench tasks this corresponds to drawing from the task's offline dataset without using scores, and on the needle-in-a-haystack tasks it corresponds to the Latin hypercube initialization that ZoMBI's authors showed fails outright when the needle occupies less than 0.05% of the search volume Fast Bayesian Optimization of Needle-in-a-Haystack Problems. Beating it is necessary but says little; it mainly calibrates task difficulty.
The strongest existing competitor, and simultaneously our most important ablation, is an online-adapted DDOM: the conditional diffusion model of Diffusion Models for Black-Box Optimization — loss reweighting toward high scorers, classifier-free guidance for extrapolation — periodically retrained on accumulated observations and sampled from pure noise at every iteration, with exploration driven purely by sampling stochasticity. DDOM earned the highest average rank (2.8) across the Design-Bench suite against seven major baselines in its original offline evaluation, so its architecture is the right thing to inherit. To make the comparison fair, both methods will share the same diffusion backbone, the same retraining schedule and training compute, the same query budget, and the same evaluation protocol; the only differences will be the warm start and the acquisition layer. Beating it requires either a higher best-found normalized score at 500 queries on the active Design-Bench tasks, or the same score at substantially lower wall-clock per iteration — we would regard a 30% reduction in per-iteration sampling cost at equal score (plausible given DiffSinger's 45.1% acceleration from shallow starting DiffSinger) as a win, but we will report both axes rather than pick one post hoc.
On the continuous needle-in-a-haystack tasks, the strongest published method is ZoMBI Fast Bayesian Optimization of Needle-in-a-Haystack Problems, which achieved a 400x compute speedup over standard Bayesian optimization at 1,000 iterations and found optima up to 3x more optimized than HEBO, TuRBO, and MiP-EGO in fewer than 100 experiments; we will run it as published on the Ackley, materials, and wildfire tasks and count our method as winning only if it reaches a comparably optimized optimum within a matched experiment count. ZoMBI does not apply to discrete Design-Bench tasks, which is itself a result we will document. For structured molecular tasks, the weighted-retraining pipeline of Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining is the reference for low-budget performance — it reached a penalized logP of 27.84 on ZINC250k with only 500 queries, where the prior machine-learning baseline reached 11.84 after roughly 5,000 — so on ChEMBL-style tasks we will include it with its latent Gaussian process and rank retraining intact, under the same 500-query budget. The key-ingredient ablations (full-diffusion start; best-seen warm start; uniform training weights) complete the baseline set and are described above with the decisive checks. No reported result found on Lacuna for per-task normalized scores of any method under an active-query version of Design-Bench, so our active-protocol numbers will be established from scratch, with the offline DDOM numbers as the anchor.
Why not just…?
Why not just run standard Gaussian-process Bayesian optimization? Because its failure mode on exactly our target landscapes is documented: stationary kernels average across narrow peaks and treat them as noise, which is why TuRBO, HEBO, and MiP-EGO all failed to find the 6D Ackley needle minimum that ZoMBI found within 900 experiments Fast Bayesian Optimization of Needle-in-a-Haystack Problems, and why exact GP inference hits its $O(n^3)$ wall beyond a few thousand iterations Return of the Latent Space COWBOYS. We keep a small GP inside our method as the scorer, but candidate generation is the diffusion model's job, so the burden of representing sharp optima is not on the GP. That said, GP-UCB retains proven sublinear regret Gaussian Process Optimization in the Bandit Setting, and we will report it on every task; if it wins on the wildfire or materials tasks where the objective may be broad rather than needle-like, we will say so.
Why not just do gradient ascent on a neural proxy, or run CMA-ES? These are cheaper than anything we propose. The Design-Bench authors found that normalized gradient ascent and CMA-ES matched or beat sophisticated generative models on 4 of 8 offline tasks Design-Bench, so this is a real threat, and we will run whitened gradient ascent on a proxy as a baseline everywhere. But the same study and DDOM both document its failure mode: gradient ascent seeks out adversarial holes — regions outside the data support where the proxy over-predicts — and in high dimensions it collapses. CMA-ES is distribution-free but population-hungry; in the sub-500-query regime that Sample-Efficient Optimization in the Latent Space targets, it cannot infer structure in unvisited space. If gradient ascent beats us on Superconductor in the active setting, we will report it as a boundary of the method.
Why not just retrain DDOM periodically and sample online? That is our strongest baseline, and we do not argue around it — we run it, compute-matched, as the full-diffusion-start ablation. The honest case for expecting it to lose is twofold: it pays full reverse trajectories for every candidate (DiffSinger's 45.1% acceleration from shallow starting is the analogous evidence DiffSinger), and it has no explicit uncertainty signal for exploration, relying on sampling stochasticity alone, whereas our batch-disagreement acquisition gives a direct epistemic readout. But the burden of proof is on us: if online DDOM matches shallow starts at equal wall-clock and equal score, the proposal's central claim is wrong, and the paper becomes a negative result about warm starts in active optimization.
Why not just warm-start diffusion from the best design seen so far, skipping the regressor? This is the ablation that most cleanly isolates our claimed mechanism, and we flag it as genuinely uncertain. The best-seen point is guaranteed valid and high-scoring, which the regressor's average is not. Our hypothesis is that the skeleton wins because it is conditioned on the extrapolated target and averages across multiple good modes, placing the diffusion start near the intersection with the trajectory of better, unseen designs, while the best-seen point anchors sampling to a neighborhood of an already-evaluated design and turns diffusion into local perturbation. The head-to-head on all tasks settles this either way; if best-seen warm starts match, the contribution narrows to "cheap warm starts for online diffusion optimization," still useful but a smaller claim.
Why not just train a better or bigger regressor and dispel the over-smoothing with capacity? Because the audio literature already ran that experiment. Beyond Oversmoothing scaled regression-based generation from 100 to 5,000 hours of LibriHeavy and found the mean-squared-error model plateaued at a Word Error Rate Ratio of roughly 2.4–2.5 — more data made it average more precisely, not diversely — while the diffusion model scaled to 1.46. Conditional-mean collapse is an objective pathology, not a capacity deficit; that is the whole reason the skeleton must be refined rather than improved.
Closing Checks
The proposal succeeds if, on at least the Superconductor and TFBind8 active tasks and the 6D Ackley needle suite, shallow-diffusion Bayesian optimization strictly dominates compute-matched online DDOM on the score-versus-wall-clock frontier within 500 and 900 queries respectively, and if the skeleton-versus-best-seen ablation confirms the trajectory-intersection mechanism rather than mere warm-starting. It fails informatively if the full-start baseline matches: that would establish that DiffSinger's phenomenon does not transfer from unconditional acoustic generation to objective-conditioned design search — a result worth publishing, since it would bound how far the two communities' shared pathology carries a shared cure.
References
- DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism
- Diffusion Models for Black-Box Optimization
- Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining
- Return of the Latent Space COWBOYS: Re-thinking the use of VAEs for Bayesian Optimisation of Structured Spaces
- Bayesian Speech synthesizers Can Learn from Multiple Teachers
- Beyond Oversmoothing: Evaluating DDPM and MSE for Scalable Speech Synthesis in ASR
- Towards Robust FastSpeech 2 by Modelling Residual Multimodality
- Optimising Hearing Aid Fittings for Speech in Noise with a Differentiable Hearing Loss Model
- Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization
- Fast Bayesian Optimization of Needle-in-a-Haystack Problems using Zooming Memory-Based Initialization
- Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design