Spatial Scaffolds That Fade: Testing Whether Synthesized 3D Speech Helps Listeners Learn or Only Helps Them Listen

Learning to follow a conversation in a second language is hard partly because the voices overlap and blur together. Binaural audio is two-channel sound that mimics what each ear actually receives. It can place each speaker at a different point in space, and that is known to make overlapping speech easier to understand. Recent work shows such audio can now be generated "zero-shot" from any ordinary single-channel recording, with no paired 3D training data, so any existing listening lesson could be spatialized for free. The open question is whether this helps people learn. A cue that makes practice easier can also become a crutch, and the learner may fail when the cue is gone: in a noisy café, on a phone call, or in a single-speaker recording.

This note proposes a semester-length study with delayed follow-up in which second-language learners practice multi-talker listening with synthesized spatial separation. The separation is either held fixed or gradually reduced, with the reduction driven by each learner's performance, until the voices come from the same place. The study measures both the easy short-term signal (in-session comprehension) and the outcome that matters: comprehension of co-located, real-room speech three and six months later. The bet is that fixed spatialization will look best on the short-term measure and worst on the long-term one, and that fading the cue will keep the early benefit without the later cost. That pattern would be a concrete, measurable instance of the "proxy trap" that education research keeps warning about, here applied to audio rendering quality.

Core Claim

Zero-shot binaural synthesis turns the location of each speaker into a free, continuous parameter that can be applied to any mono teaching material. The research question is therefore no longer "does 3D audio sound better", which perceptual scores already answer, but "how should spatial separation be scheduled over weeks of practice so that it transfers." We claim three things. First, fixed spatial separation will raise immediate comprehension but will not improve, and may impair, delayed comprehension of co-located speech. Second, separation faded adaptively to zero will match or beat plain mono practice on the delayed, co-located test. Third, the perceptual naturalness added by a neural vocoder matters less to this outcome than the schedule itself.

We expect the third claim may come out against the more expensive synthesis method, and the design is built to report that result cleanly rather than avoid it.

Grounding in Prior Work

The synthesis tool comes from Zero-Shot Mono-to-Binaural Speech Synthesis (ZeroBAS). It shifts a mono signal into two ear channels using the arrival-time difference implied by source and ear positions, scales each channel's loudness by the inverse-square law, and then cleans up artifacts with WaveFit, a vocoder pretrained on 60,000 hours of mono speech. It reports a mean opinion score (MOS, a 1–5 listener rating) of 4.07 on the Binaural Speech Dataset, matching supervised systems. On the unseen TUT Mono-to-Binaural set it scores 3.98 against 3.79 for the best supervised model, NFS. We reuse the pipeline because its source geometry is an explicit input, which makes a per-item separation schedule trivial to implement. Its evaluation stops at MOS, and our study measures learning instead.

Two neural acoustic field methods model room acoustics and personal head filtering that ZeroBAS ignores, but they are too slow to train per scene for lesson-scale content. Acoustic Volume Rendering reaches 4.71/5 realism against 1.42 for NAF. INFER needs about 24 GPU-hours per scene. Both therefore act as constraints on our design rather than tools in it.

The work that motivates and constrains the transfer test is Zhang et al. on RIR noise floors. A room impulse response (RIR) is the recorded acoustic signature of a room. That study showed that spatial release from masking (SRM, the intelligibility gain from separating target and competing voices in space) is reproduced faithfully only when the RIRs used have low noise floors, especially at 5 m distances. We therefore use only low-noise-floor measured RIRs for the real-room transfer condition. This study is also the closest link between spatial audio and human outcomes in the file, but it measures single-sentence transcription in one sitting and never measures learning.

The longitudinal side supplies the motivation. Four papers show that immediate measures can diverge from durable skill. Pechenizkiy & Gasevic observe that learning analytics rarely look beyond one term. Devi et al. find that students' confidence gains do not predict their later research output. Zhang's music-learning system notes that a high AUC on a recommendation benchmark does not by itself mean faster skill acquisition. TACT notes that the quality of a tutor's next reply is not the same as the student's eventual mastery.

None of these studies tests a perceptual aid that is designed to help in the moment, which is exactly where the gap between short-term and long-term outcomes should be largest. The adaptive fading rule borrows the closed-loop idea from Li et al.'s feedback strategy, where past interventions shape the next one. We drive that loop with a measured comprehension score rather than with behavior classification accuracy.

The closest existing connection, per the file, is no system at all: no paper spatializes instructional audio and tracks learning over time.

Research Program

We will build a lesson renderer that takes mono multi-talker dialogue and a separation angle per item, and outputs binaural audio over headphones. Two renderers sit behind it: full ZeroBAS, and the same time-shift and loudness-scaling steps with the vocoder cleanup removed. The source material is mono, two- and three-speaker English dialogues assembled for intermediate learners, with overlap and NOISEX-92 babble added at controlled levels.

The faded condition follows a per-learner staircase. After each practice block $k$, the separation angle is updated as

$$\theta_{k+1} = \max\big(0,\ \theta_k - \delta\cdot \mathbb{1}[a_k \ge \tau] + \delta\cdot \mathbb{1}[a_k < \tau - m]\big),$$

where $\theta_k$ is the azimuth separation in degrees, $a_k$ is the learner's comprehension accuracy on block $k$, $\tau$ is a target accuracy, $m$ is a tolerance band, and $\delta$ is the step size. The rule removes the spatial cue only once the learner can do without it. It mirrors the adaptive-difficulty logic of psychophysical staircases.

Participants practice three times a week for eight weeks. Every arm sees identical dialogues and comprehension questions, and only the rendering differs. Outcomes are collected at four points: in-session accuracy (the proxy), an end-of-course test, a 3-month delayed test, and a 6-month delayed test. Every test after the course uses co-located voices only, delivered in two forms: mono, and played through measured low-noise-floor room responses. Listening effort is measured by reaction time on a secondary task performed during listening, because a benefit in effort could exist even where accuracy shows none.

We will pre-register the primary contrast: the arm-by-time interaction between in-session accuracy and 6-month co-located accuracy. We target about 60 learners per arm and over-recruit to absorb expected attrition. Missing follow-ups are handled with mixed-effects models and reported per arm, so that dropout patterns that differ by arm are visible.

Baselines

The naive reference is plain mono practice: the same dialogues played identically to both ears. It is the "do nothing" condition and the format of almost all existing listening material. Any spatial arm must beat it on the 6-month co-located test to count as useful for teaching. Merely beating it in-session does not count, because that is the proxy.

The strongest competitor is fixed ZeroBAS spatialization, with speakers held at a constant ±60° separation throughout. It uses the best zero-shot renderer in the file: MOS 4.07 on the Binaural Speech Dataset and 3.98 on TUT Mono-to-Binaural, against 3.79 for supervised NFS, as reported by ZeroBAS. It is also the condition an ed-tech team would actually ship. It runs on the same dialogues, the same session count, and the same renderer as the faded arm, so the only difference is the schedule. The faded arm beats it if it is non-inferior on in-session accuracy over the first two weeks and significantly better on 6-month co-located accuracy. There is no reported result found on Lacuna for the effect of binaural rendering on delayed learning outcomes, so these comparisons establish the first such numbers.

The ablation that isolates the synthesis ingredient is faded separation with vocoder cleanup removed. It uses only ZeroBAS's time-warp and amplitude stages, which is essentially classic signal-processing panning, run on the identical staircase. If it matches full-ZeroBAS fading within a pre-registered equivalence margin, the conclusion is that the schedule, not neural naturalness, carries the effect.

A fifth arm, mono plus on-screen speaker labels, tests whether any cue that separates the voices works equally well, whether or not it is acoustic. It is given the same fading treatment: labels are withdrawn by the same staircase.

Why not just…?

Why not just use simple delay-and-gain panning instead of a neural vocoder? This is a real threat, and the design treats it as a hypothesis rather than a dismissal. ZeroBAS argues that warping without cleanup leaves artifacts, and Acoustic Volume Rendering shows that crude time shifting degrades phase and realism. Both findings, however, concern perceived quality, not learning. Over eight weeks, artifacts may only add listening effort, or they may not matter at all. The panning ablation, run on the same staircase, settles the question directly. If panning ties, we will report that cheap rendering suffices and that the schedule is the contribution.

Why not just measure an end-of-course post-test? Every arm is tested at the end of the course, so this alternative is already built into the design. What the design adds is a test of whether the end-of-course ranking of arms predicts the 6-month ranking. Prior work suggests it often does not: in Devi et al., short-term confidence failed to predict later output, and in TACT, turn-level quality is explicitly not mastery. If the rankings agree, we will say so, and future studies of spatial audio can stop at the post-test. The 6-month horizon is a deliberate compromise between the multi-year tracking the education literature calls for and what attrition makes feasible.

Why not just use real binaural recordings or supervised mono-to-binaural models? Real dummy-head recordings fix speaker positions at recording time, so they cannot be faded per learner. Supervised models fall off on unseen rooms (3.79 against 3.98 for ZeroBAS on TUT Mono-to-Binaural). The fading mechanism needs arbitrary, per-item control of geometry over existing mono content, and only the zero-shot pipeline provides that cheaply.

Why not just show who is speaking on screen? This is the most plausible non-acoustic alternative, and it is a full arm in the study. Spatial separation might help through a perceptual route, freeing up limited listening capacity, whereas labels help only by identifying who is talking. Even so, labels could produce the same pattern of short-term help and long-term crutch. If faded labels equal faded spatialization, the finding generalizes to "fade separation cues of any kind," and the case specifically for audio weakens. We would report that outcome as it stands.

Proposed Experiments and Decisive Checks

The first check is that the manipulation works. In session one, every learner completes BKB sentences from the SPANZ corpus mixed with NOISEX-92 babble, with target and masker either co-located or separated. Both spatial renderers must produce measurable spatial release from masking, replicating the effect characterized in Zhang et al.. If they do not, the study stops, since no learning hypothesis can be tested on a cue that was never perceived.

The second check is the dissociation. Fixed spatialization should beat mono in-session and tie or lose to mono at 6 months on co-located speech. If fixed spatialization wins at both time points, the crutch hypothesis is false. Spatial audio would then simply help, and the fading machinery would be unnecessary. That would be a clean and publishable negative result for our mechanism.

The third check is the core claim. Faded spatialization should beat both mono and fixed spatialization at 6 months on mono and real-room co-located tests, with the listening-effort measure reported alongside accuracy.

The fourth check is the equivalence test between ZeroBAS fading and panning fading, which determines whether neural synthesis earns its compute cost in a teaching setting.

Together these checks determine whether zero-shot 3D speech should be used in teaching at all, whether it should be used temporarily and then removed, or whether it is unnecessary for this purpose.