Many audio systems throw away phase. They store, predict or enhance only the spectrogram magnitude, which is the energy in each time-frequency cell, and then have to recover a waveform from it. Mathematicians have shown that on any regular time-frequency grid this recovery is never fully determined. Some distinct signals always share the same magnitudes, and such signals lie arbitrarily close to every signal. The known fix is to measure with several analysis windows instead of one, but that multiplies cost by 4 to 40.
We propose to measure where these near-ambiguities fall in frequency for real speech, and then to spend extra windows only where the ambiguities are both present and audible. We test whether a budgeted, band-importance-weighted multi-window measurement matches full multi-window reconstruction quality at a fraction of the cost. The yardstick is how stable the reconstruction is in a hearing-weighted sense, not whether the signal is uniquely determined.
In plain terms, phase retrieval is unstable in some directions. The listener cares about only some of those directions, so the extra measurements should target those. The contribution is a way to measure "instability that is audible" and a window-allocation scheme that minimizes it under a fixed coefficient budget.
Why this is the right question
The theory side supplies the motivation. On Foundational Discretization Barriers in STFT Phase Retrieval and Non-uniqueness theory in sampled STFT phase retrieval prove that the map from a signal to its short-time Fourier transform (STFT) magnitudes on any lattice is non-injective for any window. Denser sampling does not help, because the ambiguity comes from the lattice's parallel-line structure. Uncovering the limits of uniqueness in sampled Gabor phase retrieval shows that the ambiguous signals are dense.
These results live in continuous, infinite-length signal space. A finite, discretely sampled audio clip is a different object, and a skeptic can fairly say that practical signals may be determined uniquely. What transfers to practice is instability, not non-uniqueness. On the connection between uniqueness from samples and stability in Gabor phase retrieval and From completeness of discrete translates to phaseless sampling of the short-time Fourier transform show that uniqueness and stability come apart. A density of near-counterexamples is exactly what produces poorly conditioned directions in a finite problem. We therefore frame everything in terms of local conditioning.
The fixes fall into two groups. Multi-window STFT phase retrieval: Lattice uniqueness proves that four Hermite-derived windows restore uniqueness on lattices, though as an existence result with no algorithm. Multi-Window Approaches for Direct and Stable STFT Phase Retrieval builds stable direct inversion. On a synthetic chirp, its relative error falls from 0.7928 with one Gaussian window to 0.0084 with 40 fractionally-rotated Gaussian windows. The windows in both works are chosen to cover the time-frequency plane uniformly in energy, with no regard to hearing. We reuse their window families as our candidate pool.
The perceptual side constrains which errors matter. Band importance functions weight speech frequencies by their contribution to intelligibility and concentrate weight in roughly 400–4400 Hz. Perceptual Contrast Stretching on Target Feature for Speech Enhancement uses them to reshape training targets, and Exploiting Consistency-Preserving Loss and Perceptual Contrast Stretching to Boost SSL-based Speech Enhancement combines them with a loss that forces the predicted spectrogram to be one some waveform could actually produce.
The closest bridge is BSP-MPNet, which applies contrast stretching to both magnitude and phase inside a neural enhancer on a standard grid. All of these apply perceptual weighting to losses or features. None uses it to decide what to measure, and none asks where the phase-retrieval instability lives.
Core Claim
Let $A(f)$ be the stacked STFT magnitudes of signal $f$ under a chosen set of windows and sampled bins. The quantity we care about is the perceptually weighted local stability constant:
$$\kappa_W(f) = \min_{u \perp \mathrm{i}f,\ u \neq 0} \frac{\|J_f\, u\|_2}{\|W u\|_2}.$$
Here $J_f$ is the Jacobian of $A$ at $f$. The direction $\mathrm{i}f$, a global phase change, is excluded because it is inaudible and unrecoverable. $W$ is a linear filter whose squared frequency response equals the band importance function. A small $\kappa_W$ means some audible, band-weighted change to the signal barely moves the measured magnitudes. That is precisely the direction in which a reconstruction algorithm can drift and produce audible error.
We claim three things. First, for real speech on a standard single-window grid, the weakest directions of the unweighted constant are not spread evenly but cluster in identifiable time-frequency regions. Second, we can choose which extra windows, and which of their frequency bins, to add so as to maximize the worst-case $\kappa_W$ over a speech corpus under a fixed total coefficient budget. Third, the resulting band-allocated measurement reaches the perceptual quality of uniform multi-window measurement at far lower cost.
This differs from the closest prior work in its objective. Alaifari and Yang maximize uniform coverage of the time-frequency plane, and BSP-MPNet weights features perceptually on a fixed grid. We weight the measurement design itself.
Research Program
Stage one: the diagnostic. For 100 TIMIT utterances at 16 kHz with a 32 ms Hann window and 75% overlap, the setting of Beyond Griffin-Lim, we compute the smallest singular vectors of $J_f$ frame-block by frame-block using a matrix-free iterative method. We then record the frequency distribution of their energy. We also compute $\kappa_W$ against the unweighted constant.
This stage can kill the project cheaply. If the weak directions sit mostly outside the important band, the answer to "are lattice near-ambiguities audible?" is largely no. The allocation idea would then be moot for speech, and we would report that as the finding. The same stage probes the continuous-theory counterexamples by discretizing a few of the Grohs–Liehr constructions and measuring their band-weighted difference.
Stage two: allocation. The candidate pool is the Hermite "Onion" and fractionally rotated Gaussian "Daffodil" windows from Alaifari and Yang. A design is a selection of pairs (window, frequency band), and its cost is the number of retained complex coefficients. We choose designs greedily. At each step we add the window-band block that most increases the corpus-averaged lower quantile of $\kappa_W$ per coefficient, estimated on a training split. Once the extra windows are sampled only in some bands, the direct inversion formula of Alaifari and Yang no longer applies, and the four-window uniqueness theorem gives no guarantee. We say so plainly.
Stage three: reconstruction. For reconstruction we generalize the hybrid algorithm of Beyond Griffin-Lim to stacked frames: about 60 Difference Map iterations followed by Fast Griffin-Lim. In the stacked version, the consistency step projects onto the range of the combined multi-window analysis operator using its canonical dual frame. The magnitude step is applied only to the sampled coefficients.
Baselines
The naive reference point is single-window, 75%-overlap measurement reconstructed with the Difference Map plus Fast Griffin-Lim hybrid of Peer et al. The same pipeline is also run at 87.5% and 93.75% overlap so that its coefficient count matches each budget. This is the strongest simple competitor, because practitioners would add redundancy this way, and in finite dimensions it may be enough. The paper reports no absolute PESQ figure; the file states "no reported absolute number found," and it says only that the hybrid beat pure Fast Griffin-Lim and pure RAAR. We therefore rerun it ourselves with identical iteration counts.
The strongest existing multi-window method is the Alaifari–Yang system at full scale: 40 fractionally rotated Gaussians with direct inversion, plus the same windows run through our iterative solver. On their chirp benchmark, their reported relative error is 0.0084 with 40 windows. We first reproduce that number before trusting our implementation.
The key ablation is uniform multi-window measurement at the same coefficient budget as the allocated design. That means the same number of extra coefficients, but windows chosen by uniform $\kappa$ or spread across all bands. This ablation isolates whether perceptual allocation, rather than extra redundancy as such, is what helps.
A second ablation replaces band importance with a mel-style or equivalent-rectangular-bandwidth weighting, a frequency spacing matched to cochlear filter widths. This tests whether the specific intelligibility curve matters.
All methods share signals, iteration counts, initialization and the magnitude noise realizations. Beating the uniform ablation means a statistically significant PESQ and STOI gain at equal budget across noise levels. Matching the full 40-window system means staying within its PESQ, and within listening-test confidence intervals, at a quarter of the coefficients or fewer.
Why not just…?
Why not just oversample the single-window grid? The theorems say lattice density cannot remove the continuous-domain ambiguity. For finite discrete speech that argument is not decisive, and this is the most serious threat to the proposal. Stage one measures whether $\kappa_W$ improves meaningfully as overlap rises. The budget-matched oversampled baseline then settles the question empirically. If it matches our allocation, we will report that redundancy, not window diversity, is what matters.
Why not just weight the Griffin-Lim magnitude projection or the loss by band importance? Weighting changes which errors the solver prefers, but it cannot create information that the measurements lack. In a poorly conditioned direction, the magnitudes do not constrain the solution, whatever the weights. We run a band-importance-weighted Difference Map and Fast Griffin-Lim variant on single-window data as an additional baseline. If it closes the gap, measurement design is unnecessary.
Why not use uniform multi-window measurement with fewer windows? This is the equal-budget ablation described above. Alaifari and Yang's coverage argument predicts it should be competitive. Our claim fails if it is.
Why not skip phase retrieval with a time-domain or neural vocoder? Learned models sidestep the question rather than answer it, and they carry the mean-seeking blur noted in Geometric Iterative Retrieval for Neural Audio Codec Resynthesis. Our question concerns what magnitude measurements determine. We include one downstream check, described below, rather than competing on vocoder quality.
Proposed Experiments
The first test uses oracle magnitudes. We use TIMIT (the 100-utterance protocol) and the synthetic chirps of Alaifari and Yang. Magnitudes are perturbed with Gaussian noise at 40, 30 and 20 dB SNR. We report PESQ, STOI, log-spectral distance, relative $L^2$ error after global-phase alignment, and band-weighted error, each plotted against coefficient budget. The decisive result is a curve on which the allocated design dominates both the uniform ablation and budget-matched oversampling in PESQ and weighted error.
The second test perturbs magnitudes realistically. A contrast-stretching-trained enhancer predicts magnitudes for each window set on VoiceBank-DEMAND, and each design reconstructs from these imperfect magnitudes. For context, the published full enhancement systems report PESQ of 3.54 (PCS-CS-WavLM) and 3.65 (BSP-MPNet). These are not directly comparable to our phase-retrieval-only comparison. Our comparison holds the enhancer fixed and varies only the measurement set and phase reconstruction, including a "noisy phase" reference.
Because Geometric Iterative Retrieval found that spectral distance can disagree with listeners, a blinded listening test on 30 utterances compares allocated, uniform and single-window reconstructions at equal budget. The listening test serves as the final arbiter.
The project succeeds if weak directions concentrate in audible regions, if allocation beats equal-budget uniform and oversampled baselines on both objective metrics and listeners, and if it approaches 40-window quality at a quarter of the cost. It fails, informatively, if stage one shows the instability is inaudible or if oversampling alone matches it.
References
- On Foundational Discretization Barriers in STFT Phase Retrieval
- Non-Uniqueness Theory in Sampled STFT Phase Retrieval
- Uncovering the limits of uniqueness in sampled Gabor phase retrieval: A dense set of counterexamples in $L^2(\mathbb{R})$
- On the connection between uniqueness from samples and stability in Gabor phase retrieval
- From completeness of discrete translates to phaseless sampling of the short-time Fourier transform
- Multi-window STFT phase retrieval: Lattice uniqueness
- Multi-Window Approaches for Direct and Stable STFT Phase Retrieval
- Perceptual Contrast Stretching on Target Feature for Speech Enhancement
- Exploiting Consistency-Preserving Loss and Perceptual Contrast Stretching to Boost SSL-Based Speech Enhancement
- Magnitude-Phase Dual-Path Speech Enhancement Network based on Self-Supervised Embedding and Perceptual Contrast Stretch Boosting
- Beyond Griffin-LIM: Improved Iterative Phase Retrieval for Speech
- Geometric Iterative Retrieval for Neural Audio Codec Resynthesis