Speech recognition (speech to text) and speech synthesis (text to speech) are roughly inverse functions. Today they are built and trained separately, or else forced to share one network, which is known to hurt synthesis. The neuroscience-inspired literature on learning without backpropagation has spent a decade on a closely related question: how a dedicated top-down network can send useful learning signals to a bottom-up network without copying its weights. We propose that, in a model trained only on unpaired audio and unpaired text, the text-to-speech decoder is that top-down network. Here is the idea in two sentences. A recognizer learns to turn speech into phoneme sequences that a discriminator cannot tell apart from real text. Instead of sending that error back through the recognizer's own transposed weights, a synthesizer that maps phonemes back toward speech converts the corrected output into layer-by-layer targets, and the same synthesizer, once trained, speaks.
This matters for three reasons. First, low-resource languages have audio and some text but no transcripts, and they need both recognition and synthesis. Second, the main open problem in joint recognition and synthesis is an asymmetry: recognition compresses many audio frames into few symbols, while synthesis expands few symbols into many frames. Third, the backprop-free learning literature has a precise criterion for when a top-down network carries good credit, but that criterion has never been tested outside static classification. Our proposal couples the two directions through that criterion rather than through shared weights, and it treats the many-to-one non-invertibility of speech recognition as a hard test case for backprop-free training.
Core Claim
Two background terms are needed. Adversarial alignment trains a generator, here the recognizer, to emit phoneme sequences whose statistics match an independent text corpus, using a discriminator as critic. This is the approach of Completely Unsupervised Phoneme Recognition by Adversarially Learning Mapping Relationships from Audio Embeddings and of wav2vec-U, which applies it to clustered segments of self-supervised wav2vec 2.0 features. The weight transport problem is the observation that backpropagation needs each layer's transposed forward weights $W^T$ in the backward pass (Baldi et al.). Methods that avoid this need instead use fixed random feedback, as in Direct Feedback Alignment, or learned feedback, as in Difference Target Propagation.
The method we reuse comes from Towards Scaling Difference Target Propagation by Learning Backprop Targets. That work showed that target propagation fails when the feedback network merely learns to invert a non-invertible forward map. It succeeds when the feedback network's Jacobian approximates the transpose of the forward Jacobian, which the authors call the Jacobian Matching Condition. Their local difference reconstruction loss, which probes each layer with noise, trains feedback to satisfy this condition, and with it they match backpropagation on CIFAR-10 (89.38% versus 89.07%).
Our claim is that the same loss resolves the speech recognition/synthesis asymmetry. Plain cycle consistency, which asks synthesis to invert recognition, is ill-posed precisely because recognition is many-to-one. Jacobian matching asks only that synthesis carry recognition's local sensitivities backwards, which is well-defined for many-to-one maps. Concretely, let the recognizer have layers $h_{l+1}=f_l(h_l)$ from segment features $h_0$ to phoneme posteriors $h_L$. Let the synthesizer have layers $g_l$ mapping layer $l+1$ back to layer $l$. Hidden targets are computed as
$$\hat h_l = h_l + g_l(\hat h_{l+1}) - g_l(h_{l+1}),$$
where $\hat h_L = h_L - \eta\,\partial \mathcal{L}_{\text{adv}}/\partial h_L$ is the top target from the discriminator loss $\mathcal{L}_{\text{adv}}$ with step size $\eta$. Each $f_l$ is then updated locally toward $\hat h_l$. Each $g_l$ is trained with the local difference reconstruction loss, so that $\partial g_l \approx (\partial f_l)^T$, plus a speech-only reconstruction term that lets the bottom of $g$ produce mel-spectrograms for a vocoder.
We are explicit about one limit. The top target needs the discriminator's gradient with respect to its input. We keep ordinary backpropagation inside the discriminator, so our claim of "no weight transport" covers the recognizer–synthesizer stack, not the critic.
Research Program
The system starts from the wav2vec-U recipe to keep everything else fixed: frozen wav2vec 2.0 Large features, k-means segmentation with merging, a phonemized unpaired text corpus, and a convolutional discriminator. Inside that recipe we replace the generator's training signal. The synthesizer adds a duration predictor trained on segment lengths, which come from audio alone, and a final feature-to-mel layer trained on audio alone, followed by a HiFi-GAN vocoder. No paired data enters at any point.
We will build three variants. They separate the coupling idea from the backprop-free idea, because a skeptic should be able to accept one without the other.
Variant A is fully backpropagated. The recognizer receives ordinary adversarial gradients, and the synthesizer is trained with the Jacobian-matching loss as a coupling regularizer.
Variant B is fully target-propagated. The recognizer is updated only from synthesizer-delivered targets.
Variant C is a hybrid. The synthesizer's feedback matrices are initialized from the recognizer's weights by SVD, following Feedback-Weight Matching, which made direct feedback alignment work for fine-tuning pre-trained BERT (STSB correlation 0.76 versus 0.10 for plain feedback alignment). This variant tests whether a structured start removes the slow inner loop that makes target propagation up to 30 times slower than backpropagation.
For each variant we measure, layer by layer and throughout training, the cosine between the target-propagation update and the true backpropagation gradient. This tells us whether the synthesizer is actually carrying credit or merely acting as a regularizer. We also measure the "anti-regularization" phase predicted for feedback alignment in deep linear analyses, in which forward weights first fit noise before fitting signal. We expect this phase to interact with discriminator overfitting, which is a known source of the run-to-run instability in unpaired speech recognition.
Relation to Prior Work
Unified models motivate the coupling but constrain how it can be done. T2V2 shares one Conformer across recognition and synthesis and extracts alignments from its own recognition outputs. A Non-autoregressive Model for Joint STT and TTS shares a multimodal encoder and reports that naive joint training degrades synthesis unless it is padded with auxiliary tasks. Both require paired data, and both couple the two tasks through shared weights, which our design does not use.
Unpaired joint work leaves the loop open. Simple and Effective Unsupervised Speech Synthesis trains a separate synthesizer on wav2vec-U 2.0 pseudo-labels, so synthesis never informs recognition. SeqRQ-AE needs minutes of paired data. Cycle-consistency training bootstraps from 100 hours of paired data and reconstructs encoder states rather than audio. That paper also reports that naive 1-best pseudo-labeling diverged to 89.3% WER, which motivates a closed loop but also warns that loops can collapse.
REBORN improves segmentation with reinforcement learning. That improvement is orthogonal to ours and could be combined later. The bidirectional perception–recall network in BSD is the closest conceptual relative, a top-down "recall" path that trains a bottom-up path without weight transport. However, it has only been tested on vision benchmarks and uses spiking neurons with a contrastive loss, not adversarial targets on sequences.
Baselines
The naive reference point is wav2vec-U with its standard generator training and no synthesis coupling, run on our exact features, segmentation and text. It reports TIMIT PER 11.3 and LibriSpeech test-other WER 5.9 after HMM self-training (wav2vec-U). We compare primarily at the adversarial stage, before self-training, because that stage is where our change acts, and we report post-self-training numbers for completeness.
The strongest existing recognition method in the file is REBORN, which reports LibriSpeech test-clean PER 5.4% and WER 9.6% with HMM self-training, and TIMIT PER 12.4%. Because REBORN changes segmentation, which we hold fixed, a fair comparison either uses REBORN's segmenter under our training signal or reports both against the wav2vec-U recipe. We will do the former where compute allows.
The strongest synthesis competitor is the two-stage pipeline of Simple and Effective Unsupervised Speech Synthesis, which reports LJSpeech MOS 3.91 (supervised 3.94, real speech 4.05) and intelligibility WER 21.7% (supervised 19.2%). We re-run it with the same vocoder, text corpus and synthesizer size.
The ablation that isolates our key ingredient replaces the Jacobian-matching loss with plain cycle-consistency reconstruction under otherwise identical Variant A training. A second ablation replaces the learned synthesizer with fixed random feedback, which is plain direct feedback alignment.
All systems get the same unlabeled audio, the same text, the same hyperparameter search budget selected by the unsupervised criterion of wav2vec-U (language-model perplexity of outputs and vocabulary usage), and at least five seeds. We report the fraction of seeds that converge, since instability is a primary failure mode.
We would consider the proposal successful under any of three outcomes. The first is that Variant A matches the wav2vec-U adversarial-stage PER while its synthesizer reaches intelligibility WER at or below 21.7% without a separate pseudo-label stage. The second is that Variant A or B raises the fraction of converging seeds. The third is that Variant B lands within about one PER point of Variant A, which would be the first evidence that backprop-free training survives an adversarial, many-to-one sequence task.
Why not just…?
Why not just keep backpropagation and add a cycle-consistency loss between the recognizer and a synthesizer? This is the most serious threat, and Variant A versus its cycle-consistency ablation is designed to settle it directly. Our reason to expect a gap is the target-propagation result itself: inverse-based feedback failed on non-invertible networks until the Jacobian Matching Condition replaced it (Ernoult et al.). A recognizer that collapses segments into phonemes is non-invertible in exactly that sense. If cycle consistency matches Jacobian matching on both PER and synthesis quality, we will report that the coupling claim fails. The remaining contribution would then be the narrower backprop-free result of Variant B.
Why not just use the two-stage pipeline, which already nearly matches supervised synthesis? The pipeline's 3.91 MOS is strong, and we do not expect to beat it on naturalness. Its weakness is that recognition errors are baked into synthesis with no route back, and its intelligibility trails supervised training (21.7% versus 19.2% WER). Our test is whether the closed loop narrows that intelligibility gap and improves recognition at the same time. If it does neither, the pipeline wins.
Why not just iterate pseudo-labeling, using synthesized speech as synthetic paired data to retrain the recognizer? This back-translation loop is simple and plausible. It is our third baseline, built on the pipeline's synthesizer and run for the same number of rounds. Hori et al.'s divergence under naive pseudo-labeling (cycle-consistency training) suggests the loop is fragile without a distribution-matching anchor, but their setting was supervised-bootstrapped, so we treat this as an empirical question rather than a settled one.
Why bother removing weight transport at all, given that GPUs handle transposes cheaply? For speech engineering alone, the answer would be weak. Here, the backprop-free setting is the scientific probe, not a deployment claim. It asks whether a synthesizer can carry enough credit to train a recognizer. Every result in the file on this question comes from classification or masked language modeling, and none comes from adversarial sequence learning. We report wall-clock time honestly against the known 30-times slowdown, and Variant C tests whether SVD-initialized feedback removes it.
Proposed Experiments
The primary benchmarks are TIMIT and LibriSpeech (960 hours of audio, with LibriSpeech language-model text as the unpaired text) for PER and WER, and LJSpeech for synthesis. On LJSpeech we report MOS and the intelligibility WER of an off-the-shelf recognizer applied to synthesized speech, following the pipeline paper's protocol, and we report human MOS rather than only automatic scores, given the gap seen in joint models (UTMOS 4.08 but human MOS 2.84).
There are three decisive checks. The first is mechanistic: in Variant B, the cosine between synthesizer-delivered targets and true gradients must be clearly positive in every recognizer layer. If it is near zero in lower layers, the synthesizer is not doing credit assignment and the backprop-free claim is refuted regardless of final PER. The second is the coupling test: Variant A's Jacobian-matching loss must beat cycle consistency on synthesis intelligibility at equal or better PER. The third is the stability test: across seeds, the fraction of runs that pass wav2vec-U's unsupervised selection criterion must not fall below the wav2vec-U baseline.
A positive outcome on all three would give a single unpaired model that both recognizes and speaks, in which the speaking half is what teaches the listening half.
References
- Completely Unsupervised Phoneme Recognition by Adversarially Learning Mapping Relationships from Audio Embeddings
- Unsupervised Speech Recognition
- Learning in the machine: The symmetries of the deep learning channel
- Direct Feedback Alignment Provides Learning in Deep Neural Networks
- Difference Target Propagation
- Towards Scaling Difference Target Propagation by Learning Backprop Targets
- Enabling Fine-Tuning of Direct Feedback Alignment via Feedback-Weight Matching
- Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks
- T2V2: A Unified Non-Autoregressive Model for Speech Recognition and Synthesis via Multitask Learning
- A Non-autoregressive Model for Joint STT and TTS
- Simple and Effective Unsupervised Speech Synthesis
- Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning
- Cycle-consistency Training for End-to-end Speech Recognition
- REBORN: Reinforcement-Learned Boundary Segmentation with Iterative Training for Unsupervised ASR
- Biologically Plausible Learning via Bidirectional Spike-Based Distillation