Speech-to-speech translation for languages without a writing system cannot rely on text. Instead, systems turn audio into discrete "units": labels for short stretches of sound, produced by clustering the internal features of a self-supervised speech model such as HuBERT. A translator then maps source units to target units, and a vocoder turns the target units back into audio. These units are fragile. The same sentence spoken by a different person, in a different room, or in a relaxed conversational style comes out as a different unit sequence. Current fixes are hand-built. One perturbs pitch and energy at random. Another converts every voice into one reference speaker before clustering. Neither holds up when systems trained on read or mined audio are tested on real conversation, a failure the Taiwanese Hokkien work reports explicitly. For tonal languages these fixes carry a second cost: pitch is partly speaker style and partly word meaning, so flattening it can erase the very contrasts the translator needs.
We propose to fine-tune the speech encoder that produces the units with an adversary, a small classifier trained to guess the recording condition from the encoder's features while the encoder learns to make that guess fail. The adversary sees only per-utterance summary statistics of the features, and the encoder must also keep predicting a speaker-normalized pitch contour. The working hypothesis is a simple division of labor. Whatever stays constant across an utterance (voice, pitch range, microphone, room) is nuisance and should be removed. Whatever varies within the utterance relative to its own average (phones, lexical tone) is content and must be kept. The recording conditions themselves are not labeled by hand. They are discovered by clustering the same summary statistics.
Core Claim
Our claim is that adversarial removal of utterance-level statistics, combined with an explicit constraint that keeps within-utterance relative pitch, yields unit vocabularies that transfer from read or mined speech to conversational speech better than perturbation or reference-speaker normalization. We expect the advantage to come from shifts that cannot be simulated by signal processing, such as accent, speaking style and spontaneous delivery, and to come without destroying tone in Hokkien. If the gains appear only on simulatable shifts like added noise or reverberation, stronger augmentation is the right answer and the adversary is unnecessary. The experiments are designed to separate these two outcomes.
Relation to Prior Work
Two textless translation systems motivate the proposal by showing that acoustic variation in units is the bottleneck. TranSpeech randomly perturbs source pitch, energy and speaking rate and normalizes target pitch and energy, gaining an average of 2.9 BLEU on CVSS-C. Textless Speech-to-Speech Translation on Real Data maps all speech to one reference speaker before extracting units, gaining 3.2 BLEU on VoxPopuli. Both fix the nuisance in the signal or in a separate front-end. We instead change the feature space from which units are clustered, and we let the training data decide what counts as nuisance.
Speech-to-Speech Translation for a Real-world Unwritten Language supplies the application regime and the target failure. Its two-pass UnitY system for English and Hokkien uses a Mandarin text pivot and reference-speaker normalization, and the authors report severe drops on conversational speech even with about 8.1k hours of mined audio. UWSpeech instead anchors units to phonetic transcripts from written helper languages and leaves acoustic robustness as future work. Our method needs neither a sibling written language nor phonetic transcripts, so it also applies to isolated oral languages. It can still be stacked on top of either pivot.
The method borrows from adversarial domain adaptation. The basic recipe is the encoder-versus-discriminator game with a gradient reversal layer, which flips the discriminator's gradient before it reaches the encoder, as in Ren et al., Huang et al. and FAN-Net. Two pieces of evidence shape where the adversary sits. ALIGN shows that adversarial alignment on intermediate layers removes session drift in speech decoding from brain signals, and that time-stretch augmentation alone was not enough. It cut held-out phoneme error from 52.61% to 43.39%. Domain Agnostic Few-shot Learning for Speaker Verification shows that clustering activation means and standard deviations into pseudo-domains beats human-defined noise labels. That paper keeps speaker identity as its target, whereas we treat exactly those statistics as the thing to remove. FAN-Net's finding that style lives in low-frequency Fourier amplitude while structure lives in phase is the image-domain analogue of our split between slowly varying and fast-varying information.
The risk that adversarial invariance can over-smooth features and erase task information constrains the design. For a tonal language that risk is concrete, which is why the tone constraint below is part of the method rather than an afterthought.
Research Program
We start from a pretrained multilingual HuBERT encoder and fine-tune its upper layers, call their output $h_{1:T}$, on unlabeled source and target speech drawn from many conditions. The pool includes read Common Voice, mined LibriVox and drama audio, and whatever unlabeled conversational audio exists. No translations or transcripts are used at this stage.
Pseudo-domains are built first. For each utterance we compute the mean and standard deviation of $h$ over time, plus absolute log-f0 mean and range, and cluster these vectors into $K$ groups. The clusters are refreshed periodically during training. The fine-tuning objective is
$$ \mathcal{L} = \mathcal{L}_{\text{content}} + \lambda_{\text{tone}}\,\mathcal{L}_{\text{tone}} - \lambda_{\text{adv}}\,\mathcal{L}_{D}\big(\mu(h), \sigma(h)\big). $$
The three terms are defined as follows.
$\mathcal{L}_{\text{content}}$ is the original HuBERT masked-prediction loss against the frozen model's unit targets. It anchors content and keeps the encoder from drifting into an uninformative space.
$\mathcal{L}_{\text{tone}}$ is the regression error of a small head that predicts the utterance-normalized log-f0 contour, meaning per-utterance z-scored pitch with absolute height removed, from frame features $h_t$.
$\mathcal{L}_D$ is the cross-entropy of a discriminator $D$ that predicts the pseudo-domain from the time-pooled mean $\mu(h)$ and standard deviation $\sigma(h)$. The gradient reversal layer means the encoder is pushed to raise this loss.
$\lambda_{\text{tone}}$ and $\lambda_{\text{adv}}$ are scalar weights. They are tuned on a held-out development set scored with the text-free metric BLASER, so tuning never requires target-language transcripts.
Because $D$ sees only pooled statistics, it cannot directly penalize within-utterance pitch movement. The tone head makes that protection explicit.
After fine-tuning, we re-run k-means on the new features to obtain the unit vocabulary. We then train the same unit-based HiFi-GAN vocoder and the same translators used by the comparison systems: the S2UT translator on CVSS and VoxPopuli, and two-pass UnitY with the Mandarin pivot on Hokkien. A second Hokkien variant removes the Mandarin pivot entirely, to test the no-sibling regime.
Beyond translation scores, we will analyze the units directly. We will measure how many distinct unit sequences the same sentence receives across speakers and conditions, how much tone information survives (described below), and how much speaker identity a probe can still recover from the units.
Baselines
The naive reference point is the unmodified HuBERT encoder with k-means units and no perturbation or normalization, trained with the same translator and vocoder. Every gain is reported against it.
The strongest existing method on the Hokkien question is the two-pass UnitY system with reference-speaker normalization from Speech-to-Speech Translation for a Real-world Unwritten Language. It reports 7.8 ASR-BLEU for English to Hokkien and 10.0 for Hokkien to English on TAT-S2ST, with MOS between 3.5 and 3.7. We will retrain it on identical data, including the mined portion, with the same compute and the same number of tuning runs as our method, and differ only in how units are produced. Matching 7.8 and 10.0 in-domain while beating it on conversational audio by a margin larger than seed variance (at least three seeds), on both ASR-BLEU and BLASER, counts as beating it. For conversational out-of-domain speech we found no reported result on Lacuna, so we will establish that number ourselves with the retrained baseline.
On non-tonal benchmarks, the strongest competitors are TranSpeech's Bilateral Perturbation, which reports an average of +2.9 BLEU over textless baselines on CVSS-C Fr→En, En→Es and En→Fr, and the reference-speaker normalizer, which reports +3.2 BLEU on VoxPopuli. Both will be rerun on the same splits with the same translator.
We will also add an augmentation-strong baseline: Bilateral Perturbation plus room reverberation, additive noise, codec simulation and speed perturbation. It receives the same tuning budget as our adversary weights.
Three ablations isolate the proposal's ingredients. The first replaces pseudo-domains with categorical corpus labels. The second uses a frame-level discriminator instead of pooled statistics, which we predict will damage tone. The third removes the tone head. In each case the question is whether the full method beats the ablation on Hokkien conversational speech and on the tone test.
Why not just…?
Why not just use stronger augmentation instead of an adversary? This is the most serious threat. Noise, reverberation and channel effects can be simulated, and augmentation may match an adversary on them. ALIGN found augmentation alone insufficient, but that was for neural recordings, not audio, so it does not settle the question here. We will split each out-of-domain test set into simulatable degradations (added noise and reverberation on clean speech) and non-simulatable ones (new speakers, accents, spontaneous conversation). If the augmentation-strong baseline ties us on both, we will report that the adversary is unnecessary. The claim survives only if we win on the non-simulatable split.
Why not just normalize every voice to one reference speaker? Normalization helps, with +3.2 BLEU reported in Textless Speech-to-Speech Translation on Real Data. But the normalizer is itself a model trained on clean data, and the Hokkien work shows the normalized system still breaks on conversation. Normalization also maps pitch to one speaker's range, which preserves tone only if the voice conversion gets each contour right. We will measure tone retention through the normalizer and through our encoder on the same utterances, and we will also try stacking the two methods.
Why not just mine more data? Mining helped, adding +2.0 BLEU on VoxPopuli, but over 8.1k mined Hokkien hours did not prevent the conversational collapse reported by Speech-to-Speech Translation for a Real-world Unwritten Language. Our encoder uses the same mined audio, so the comparison tests whether extra data is better used as varied acoustic conditions for the adversary than as more parallel training pairs.
Why not just anchor units to phones from helper languages? UWSpeech's phone anchor raised Fisher BLEU from 7.17 to 17.33, but it requires transcribed helper-language data, and its robustness to acoustic shift is untested. The anchor and our method address different problems: what the units mean versus how stable they are. We will test them as complements on Fisher, where the anchor is reproducible, rather than claim to replace it.
Proposed Experiments
The primary experiment is Hokkien↔English on TAT-S2ST. We test in-domain and on held-out conversational audio, report ASR-BLEU, BLASER and MOS, and run both with and without the Mandarin pivot.
The second experiment tests tone. We will score tone retention with an ABX discrimination test, which checks whether units for two different tones are farther apart than units for two tokens of the same tone, on syllable pairs that differ only in tone. Tone labels come from TAT romanization where it carries tone marks. Where no tone annotation exists, we fall back to clusters of normalized f0 contours. The decisive check is that the frame-level-discriminator ablation loses tone discrimination while the full method does not, and that the gap tracks translation quality on tone-dependent sentences.
The third experiment is the non-tonal control: CVSS-C against Bilateral Perturbation, and VoxPopuli against the normalizer, each with cross-condition evaluation on LibriVox and Common Voice speakers held out from training. Here the method should at least match prior gains.
Two analyses explain the results. One measures unit-sequence consistency across speakers for the same sentence. The other measures speaker-ID probe accuracy on the units. Both should improve before translation does. If they improve but translation does not, the bottleneck lies elsewhere, and that is a useful negative result in its own right.
References
- TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation
- Textless Speech-to-Speech Translation on Real Data
- Speech-to-Speech Translation for a Real-world Unwritten Language
- UWSpeech: Speech to Speech Translation for Unwritten Languages
- Adversarial Domain Adaptation for Classification of Prostate Histopathology Whole-Slide Images
- Domain Adaptation on Multiple Cloud Recognition From Different Types of Meteorological Satellite
- Fan-Net: Fourier-Based Adaptive Normalization for Cross-Domain Stroke Lesion Segmentation
- ALIGN: Adversarial Learning for Generalizable Speech Neuroprosthesis
- Domain Agnostic Few-shot Learning for Speaker Verification
- BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric