Speech Harmonics as a Free Frequency-Diverse Array

Voiced speech is a stack of sinusoids at whole-number multiples of one base rate, the fundamental frequency $F_0$ or pitch. Microphone-array methods that locate a talker in angle and distance usually treat this harmonic structure as a nuisance. They pick one carrier frequency, assume it is known, and suffer when it is wrong or when the array is large enough that phases wrap past $2\pi$. We propose to treat the harmonics as the main asset. Under one assumption, an array that hears all harmonics of one voice at once is a "frequency-diverse array": a radar design in which elements transmit at slightly offset frequencies. Here the offset is simply $F_0$. The assumption is that we compare the phase difference between two sensors on opposite sides of a circular array, harmonic by harmonic. That difference then rotates by a constant amount from one harmonic to the next. Estimating this rotation is a one-dimensional frequency estimate over the harmonic index. It recovers direction without the phase-wrapping ambiguity that limits single-frequency methods.

The same structure gives a cheap check on the pitch estimate itself. If the pitch hypothesis is right, every harmonic bin points to the same direction. If it is an octave wrong, half the bins are either empty or carry energy from somewhere else. The proposal is a non-iterative estimator that solves pitch and near-field location together in closed form. It falls back to an expensive refinement only on the frames where this physical consistency check fails. The goal is the accuracy of joint search or progressive enhancement, at close to the cost of a single pass.

Core Claim

Two existing literatures each assume away the other's hardest variable. Near-field localization with a uniform circular array removes the distance term by taking phase differences between diametrically opposite sensors. It then solves for angle by least squares and searches only over distance (Mixed Incoherent Far-Field and Near-Field Source Localization under Uniform Circular Array). A related method turns the circular data into a structure that can be solved algebraically (Fast BSC-based algorithm for near-field signal localization via uniform circular array). Both treat source frequency as given, and the first explicitly reports that frequency error degrades angle and range. Pitch trackers, in turn, are monophonic and single-channel. In heavy noise they either fail, as PESTO reports for its own model (PESTO), or need a chain of progressive enhancement stages (SNR-Progressive Model with Harmonic Compensation).

Our claim is that for harmonic sources these two problems are best solved together in one pass. The central quantity is the following. Take a circular array of radius $R$ and a diametric sensor pair $m$ at angle $\gamma_m$. Let $\Delta_m(h)$ be the measured pair phase difference at harmonic $h$, which lies at frequency $hF_0$. Under the near-field Fresnel approximation the second-order range term cancels in the pair difference, leaving

$$ e^{j\Delta_m(h)} = \exp\!\big(j\,h\,\omega_m\big), \qquad \omega_m = 2\pi F_0 \frac{2R}{c}\,(u\cos\gamma_m + v\sin\gamma_m), $$

Here $c$ is the speed of sound, and $(u,v)=(\sin\theta\cos\phi,\sin\theta\sin\phi)$ encodes elevation $\theta$ and azimuth $\phi$. Across harmonics, each pair's phase is a complex sinusoid in $h$. Its "frequency" $\omega_m$ can be read off by an FFT over the harmonic index, and this works even when each individual $\Delta_m(h)$ has wrapped many times. The estimate is unambiguous whenever $|\omega_m|<\pi$, which holds when $F_0 < c/(4R)$. For a 5 cm radius this bound is about 1.7 kHz, well above any speaking pitch. Given $\omega_m$ for all pairs, $(u,v)$ follows by linear least squares, exactly as in the diametric method. Distance then follows from a one-dimensional search per source.

The phase-ambiguity limit noted for fixed circular arrays (Ambiguity Resolution for Phase-Based 3-D Source Localization under Fixed Uniform Circular Array) is therefore sidestepped for harmonic sources rather than resolved after the fact. The coupling between angle and range is the problem studied for coprime frequency-diverse radars (Cramér-Rao Bound of Joint DOA-Range Estimation for Coprime Frequency Diverse Arrays). Here the diametric cancellation removes it before the harmonic-index step.

Pitch enters through a consistency test rather than a separate tracker. Under a candidate $F_0$, the bins at its odd multiples must carry energy that is spatially coherent with the even bins and points to the same $(u,v)$. A halving error ($F_0/2$) predicts harmonics at bins that are empty or point elsewhere. A doubling error ($2F_0$) leaves coherent, same-direction energy at the skipped odd multiples. Comparing a handful of candidates ($F_0/2$, $F_0$, $2F_0$, plus near neighbours) costs one coherence score each. With two talkers whose pitch tracks cross, the bins of each comb point to different directions. Direction then separates the combs that a single-channel tracker cannot tell apart (High-Concentration Time-Frequency Representation and Instantaneous Frequency Estimation of Frequency-Crossing Signals).

Research Program

The work has three parts. First, we derive and implement the harmonic-index estimator described above, including its multi-source form. With $K$ talkers, the harmonic bins of each pitch candidate are scored separately, so the closed-form angle step runs once per comb. This attacks the single-source limitation left open by the algebraic method. Second, we characterise when the closed form breaks. Expected failure points are few usable harmonics at very low signal-to-noise ratio (SNR), strong reverberation that smears inter-sensor phase, harmonics shared by two talkers at integer pitch ratios, and the Fresnel approximation at very short distances. We also derive a Cramér-Rao-style bound showing how many harmonics are needed for a target angular accuracy.

Third, we add selective refinement. On frames where the best candidate's coherence score falls below a threshold, we invoke an expensive fallback. The fallback is either a joint grid search over pitch, angle and distance, or pitch tracking on the intermediate output of an SNR-progressive enhancer. All other frames are accepted as computed.

This third part borrows a lesson from information extraction rather than from signal processing. In relation extraction, unguided self-refinement made results worse. Triggering a costly correction pass only when an external checker flags an error gave the gains at low cost (Enhancing Relation Extraction via Supervised Rationale Verification and Feedback). Lightweight, residual-only refiners such as iRe2f keep most of the gain of full re-encoding at much higher throughput. Our checker is physical, not learned: cross-harmonic directional coherence needs no labels. Open information extraction therefore enters this proposal only as a methodological lens. A speaker-attributed front end of this kind is what spoken open-extraction pipelines would eventually need, since Towards relation extraction from speech locates their bottleneck in front-end entity errors. Evaluating that downstream use is out of scope here. The broad finding that refinement gains saturate after two or three passes (Deliberation Networks; Learning with Rethinking) motivates measuring how much of that saturating gain a zero-pass estimator already captures.

Baselines

The naive reference point is the diametric circular-array method from Mixed Incoherent Far-Field and Near-Field Source Localization under Uniform Circular Array, applied as published. It uses a single dominant frequency per frame, taken from the strongest FFT peak with bin interpolation, and no pitch modelling. The paper reports 0.26 s per run against 50.95 s for two-stage MUSIC. It also reports angle error below 1 degree and range error below 0.2 m for frequency errors within 2 kHz. Those figures come from a radio-frequency simulation, so they do not transfer to acoustic scales and we will re-measure on our setup. Beating this baseline means lower angle error at equal runtime on frames where the dominant peak is a higher harmonic that has wrapped.

The strongest existing route for pitch is sequential: a dedicated tracker followed by single-frequency localization at the estimated $F_0$ or its harmonics. Two trackers are used in this cascade. PESTO (PESTO) reports about 88% raw pitch accuracy zero-shot on PTDB, against 63% for the supervised PENN baseline. The SNR-progressive model (SNR-Progressive Model with Harmonic Compensation) reports 89.9% pitch accuracy at −15 dB from its intermediate stage, against 68.1% from raw input, on the 1,000-hour DNS-3/DCASE/room-impulse-response benchmark. Both trackers run on a reference channel and on a delay-and-sum beamformed channel, and we report the better of the two. Beating them means higher raw pitch accuracy and a lower octave-error rate at the same SNR. For the SNR-progressive cascade, it also means a large reduction in compute.

The strongest localization competitor is exhaustive joint search. This is three-dimensional MUSIC over angle and distance, combined incoherently across harmonic bins, with pitch as an extra grid dimension. Fast BSC-based algorithm reports 2.12 s for 3D-MUSIC against 0.73 s for its closed-form method on a 16-sensor simulation, and that the closed form matches 3D-MUSIC accuracy only at high SNR. We also run an alternating refinement baseline with $T=1,2,3$ passes: estimate pitch, localize, beamform, re-estimate pitch. No reported result was found on Lacuna for joint pitch and near-field localization of speech on any benchmark, so all of these are run by us.

For fairness, every method sees the same simulated recordings and the same STFT front end and analysis window. All are timed on the same CPU. Every free threshold or grid resolution gets the same tuning budget on a held-out split. Three ablations isolate the key ingredients. One removes the harmonic-index FFT and estimates angle from the single strongest harmonic. One removes the coherence test over pitch candidates. One removes the selective trigger, so the fallback runs either always or never.

Why not just…?

Why not just run a good pitch tracker first and plug its output into the known-frequency array method? This is the most serious alternative. If pitch were reliably correct, the joint formulation would add only the harmonic-index unwrapping. The evidence that pitch is not reliable where it matters comes from the trackers' own reports. PESTO describes rapid degradation near 0 dB, and raw-input tracking drops to 68.1% at −15 dB in the SNR-progressive study. The sequential baseline above settles the question directly. If the cascade with the SNR-progressive tracker matches our octave-error rate and angle error, the contribution reduces to a cheaper route to the same answer. We would then report it as such.

Why not just use the low harmonics, where the array is too small for phases to wrap? For a small array this already avoids ambiguity. The cost is the loss of the higher harmonics, which carry most of the angular resolution because phase difference grows with frequency. The low harmonics are also often the most masked by low-frequency noise. We sweep array radius from 3 cm to 20 cm. If single-low-harmonic estimation matches ours at the small radii, the method's value is confined to larger apertures, and we will say so.

Why not just run joint grid search or a fixed number of alternating passes? These are the accuracy ceiling, and we expect them to equal or beat the closed form at low SNR. That is why they serve as the fallback rather than as a competitor to be dismissed. The question is the accuracy-versus-compute curve. The diminishing-returns pattern across the refinement literature suggests that most frames do not need extra passes. Our experiments measure what fraction of frames the coherence test sends to the fallback, and how much accuracy is lost on the frames it accepts.

Why not just train a neural network end to end for pitch and direction? It may learn the same structure. It would, however, need labelled multichannel data for every array geometry, and it gives no physically interpretable signal for when to refine. We include a small convolutional baseline trained on the same simulated data as a reference. It is not the focus of this proposal.

Proposed Experiments

The data are simulated circular arrays of 8 and 16 microphones at radii from 3 to 20 cm, built from PTDB speech, whose laryngograph-derived pitch serves as ground truth. Recordings are convolved with simulated room impulse responses and mixed with DNS-3 and DCASE noise at −15 to 20 dB, mirroring the SNR-progressive benchmark. Sources are placed at 0.3 to 3 m to span near and far field. We test single talkers and two-talker mixtures, including pairs selected for crossing pitch tracks and for near-integer pitch ratios. MIR-1K singing provides a check on wide pitch range and long sustained notes.

Metrics cover four things. Pitch is scored by raw pitch accuracy, octave-error rate and voicing drop-out rate. Localization is scored by angle and range RMSE, and near/far classification accuracy using the range-spectrum criterion of the mixed-source paper. For two-talker mixtures we report direction-assignment accuracy. Cost is runtime and floating-point operations per frame, plus the fraction of frames routed to the fallback.

Three checks decide the outcome. First, against the sequential cascade, the joint estimator must cut the octave-error rate at 0 dB and below without losing accuracy at high SNR. If it does not, the coupling argument fails. Second, against the known-frequency array method at radii where upper harmonics wrap, the harmonic-index estimator must reduce angle error. If it does not, the frequency-diverse-array view adds nothing. Third, with selective refinement, the system must reach within a small margin of the always-refine accuracy while invoking the fallback on a minority of frames. If coherence fails as a trigger, the cost argument fails. The main stated limitation is that all evidence is simulated. No real multichannel speech benchmark with pitch labels appears in the file, so a small real recording with a laryngograph is the natural next step.