Voice–Grammar Concordance in Expressive Speech Translation

Expressive speech-to-speech translation systems take a person's spoken sentence, translate it, and speak the translation back in that person's voice, keeping their pitch, rhythm and emotional tone. In languages like Italian or Spanish, a first-person sentence such as "I am tired" must be inflected for the speaker's gender ("stanca" versus "stanco"). English text gives no clue, so the only evidence is the voice. Today's systems get the voice right and the grammar wrong. The voice-cloning stage copies a woman's voice faithfully. The translation stage, which has lost the speaker's gender inside its internal bottleneck, picks the masculine form. The listener then hears a woman's voice calling herself "stanco." Nobody has measured how listeners perceive this mismatch, and current naturalness metrics do not look for it.

We propose to treat agreement between the rendered voice and the grammatical gender as a property of naturalness in synthesized speech, and to build a system around one principle. The grammatical choice should be conditioned on the voice the listener will actually hear, as a graded perceptual quantity, rather than on a hard label guessed from the source. The work has two parts. The first is a controlled listening study that crosses inflection with rendered voice, including acoustically gender-ambiguous voices, to find out whether and how much listeners penalize mismatch. The second is a translation model whose gender decision reads a continuous "how female/male does the output voice sound" score taken from the same speaker representation that drives synthesis. When that score is ambiguous, the model falls back to gender-neutral rewording.

Why this gap is real

Three bodies of work meet here without overlapping. On the bias side, Different Speech Translation Models Encode and Translate Speaker Gender Differently shows that the adapter in modular systems like SeamlessM4T erases speaker gender. The adapter is the layer that maps speech features into the text decoder's space. After it, probing F1 falls from 96.19 in monolithic models to about 54, and Italian female accuracy is 12.09% against 90.55% for males. FBK@IWSLT Test Suites Task: Gender Bias evaluation with MuST-SHE shows the failure is specific to cases where only the voice disambiguates. When textual cues exist, feminine accuracy is 85–87%. When only the voice is available, it is 39–56%. These papers motivate the proposal, but they evaluate text output only.

On the synthesis side, Seamless: Multilingual Expressive and Streaming Speech Translation transfers vocal style into target speech through PRETSSEL, an acoustic model that renders speech from a paralinguistic embedding without intermediate text labels. Its translation stage, however, sits behind the same lossy adapter. Its expressivity evaluation checks prosodic consistency, not grammatical agreement with the voice. TTSDS - Text-to-Speech Distribution Score establishes that prosody, not noise, now separates good from bad systems, and that MOS saturates near the top. That suggests subtler failures like concordance need dedicated measurement.

Two papers act as constraints. Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translation finds that translation models infer gender mainly from formant bands (roughly 400–2500 Hz, reflecting vocal-tract shape) rather than from pitch. It also finds that the first-person "I" is often the decisive audio segment. Prosody-controllable Gender-ambiguous Speech Synthesis finds that perceived gender ambiguity comes from timbre interpolation, not pitch shifts. Together they rule out a design that reads gender from pitch-only prosody tokens of the kind used in ProsodyLM. The perceptual score must come from the full timbre embedding.

Core Claim

In expressive speech translation, the correct target for first-person grammatical gender is the perceived gender of the output voice. Mismatch between the two is a naturalness failure that listeners detect and that MOS-style metrics underweight.

A model that conditions inflection on a graded perceptual score of the output speaker embedding has three properties. It matches a source-classifier cascade when the voice is clearly gendered and unchanged. It beats the cascade when the output voice differs from the source, for example when a user selects a voice or style transfer is partial. It also beats the cascade when the voice is ambiguous, because there a hard binary label forces an arbitrary and often jarring choice.

We say plainly that the first regime is where simple cascades are expected to suffice. The novelty lies in the measurement and in the other two regimes.

Research Program

Concordance listening study. We cross three inflection conditions with a voice continuum while holding prosody fixed. The inflection conditions are masculine, feminine, and a neutral rewording that avoids gendered forms. The voice continuum has five points, interpolated between a female and a male speaker embedding in the manner of the gender-ambiguous TTS paper but implemented in a modern expressive decoder. Prosody is held fixed by reusing the same predicted pitch and duration contours, so the only difference between conditions is the grammar and the timbre.

Stimuli come from MuST-SHE Category 1 sentences translated into Italian and Spanish. Native listeners give naturalness ratings and pairwise preferences, and separately rate perceived speaker gender. This produces a curve of naturalness penalty as a function of the gap between perceived gender and inflection. We also measure whether the neutral rewording is preferred in the ambiguous middle of the continuum.

The asymmetry reported by Modeling incompletion phenomenon in Mandarin dialog prosody shows that listeners weight some mismatches far more than others. In that paper, conversational prosody over a formal voice is accepted more readily than the reverse. That is a reason to measure this curve rather than assume it.

Perceptual voice score. We train a small regressor that maps the speaker embedding fed to the synthesis stage to the probability that listeners perceive the voice as female. It is supervised by the listener gender ratings from the study plus SGR pseudo-labels on LibriTTS-R. SGR is speech gender recognition; the layer-3 wav2vec 2.0 SGR reaches 99.1% on clean speech per the gender-ambiguous TTS paper. Because the regressor reads the embedding that will be rendered, it tracks the output voice, not the input.

Concordant translation. We fine-tune an open speech translation model with three prefix modes: masculine, feminine, and neutral rewording. The training data are balanced translations of first-person utterances generated by an LLM, following the data rectification recipe of Addressing speaker gender bias in large scale speech translation systems, extended with a neutral-rewording target. At inference, the perceptual score selects the mode through two thresholds tuned on held-out listener data. Feminine is chosen above the upper threshold, masculine below the lower, and neutral in between. A soft variant feeds the score itself as a continuous conditioning vector.

The translated text then goes to the expressive synthesizer with the same speaker embedding, so that the voice and the grammar come from the same source.

Baselines

Naive reference. The do-nothing point is SeamlessM4T v2 with SeamlessExpressive, run unmodified on our test sets. For text-level gender translation accuracy the file gives a strong prior. Addressing speaker gender bias reports SeamlessM4T feminine accuracy of 2.50% on Spanish MuST-SHE, and Different Speech Translation Models reports 12.09% female accuracy for Italian. We expect massive concordance failure for female voices. The open question is how much naturalness it costs, and no reported result was found on Lacuna for concordance-related naturalness of expressive S2ST.

Strongest existing method. This is the three-mode gender-token system of Addressing speaker gender bias, which reports 87.05% feminine accuracy on English-to-Spanish MuST-SHE and 84.72% on English-to-Italian. Its released weights are not described in the file, so we reimplement it on the same open ST backbone, with the same rectified training data and tuning budget as our model. We run it in automatic mode, using its encoder-side gender-representation loss, and pair its output with the same expressive synthesizer.

Source-classifier cascade. This is the competitor a skeptic will favor. It runs the wav2vec 2.0 SGR on the source audio and uses the hard label to pick the masculine or feminine prefix in our own fine-tuned model. Training is identical, so this baseline differs from our method only in what the gender decision reads.

Ablations. Two ablations isolate the key ingredients. The first replaces the output-embedding score with a score computed from the source embedding. That tests whether the output-conditioned design itself matters. The second replaces the full timbre embedding with pitch-only features (median, range and slope of log pitch, as in ProsodyLM's tokens). That tests the formant finding from Voice, Bias, and Coreference.

What counts as beating the baselines. All systems share backbone, data and compute. To beat the strongest baseline, our method must do two things. First, it must match the reimplemented three-mode system's feminine accuracy within noise in the unchanged clear-voice regime. Second, it must significantly exceed both the three-mode system and the cascade on concordance and naturalness preference in the voice-changed and ambiguous regimes, without losing ASR-BLEU.

Why not just…?

Why not just run a gender classifier on the source and prepend a tag? In the most common case, a clearly gendered speaker whose voice is cloned unchanged, this probably works about as well as our method. We include it as a full baseline instead of arguing around it. The tag decision has two blind spots, though. It is made on the source, so it goes stale when the output voice differs from the input. It is also binary, so for ambiguous voices it forces a coin flip that our listening study will price. Renduchintala's Investigating Failures of Automatic Translationin the Case of Unambiguous Gender further shows that untuned text models can override explicit gender cues when those cues clash with occupational stereotypes. We therefore report cascade results split by stereotype congruence. If the cascade ties our method in all three regimes, the method claim is falsified and the contribution reduces to the measurement study.

Why not just instruct an audio LLM to respect the speaker's gender? Prompted debiasing in Gender Bias in Instruction-Guided Speech Synthesis Models flipped Parler-TTS's occupational distributions rather than balancing them. Do Audio LLMs Listen or Read? reports that Audio Flamingo 3 scores 17.40% and Qwen2-Audio 14.85% on VoxParadox before mitigation, because the models follow text over audio. A Tale of Pronouns finds near-zero attribution to gender markers when priors dominate. We will still run one prompted audio LLM as a cheap check on MuST-SHE Category 1, but the evidence predicts it will fall short.

Why not just use SeamlessExpressive end to end? Its voice transfer is exactly what creates the clash. Its translation path inherits the adapter erasure documented by Different Speech Translation Models. It serves as our naive reference, and the listening study measures directly what its mismatches cost.

Why not just train on gender-balanced data, as the three-mode paper does? Balanced data fixes the masculine default for clearly gendered speakers. The three-mode baseline above captures that fix, and our model uses the same data. Balanced binary targets still give no answer for ambiguous voices and no mechanism for tracking a changed output voice, which are the regimes where we claim a gain.

Proposed Experiments

Test sets. We use MuST-SHE Category 1 and Category 2 for English to Italian and English to Spanish, restricted to language pairs covered by the expressive synthesizer. We evaluate under three regimes. In the unchanged regime, source voice equals output voice. In the voice-changed regime, the output voice is drawn from LibriTTS-R speakers of the opposite or same perceived gender. In the ambiguous regime, output voices are interpolated embeddings along the continuum.

Metrics. Gender translation accuracy is computed on ASR transcripts of the output speech. Concordance rate is the agreement between inflection and listener-perceived gender, with the neutral form counted as concordant only in the ambiguous band. We also collect human naturalness ratings and pairwise preferences, TTSDS scores (whose reported correlation with human judgment is 0.60–0.83, and which needs 80–100 samples per system), ASR-BLEU, and prosodic consistency judgments following the Seamless protocol. For Category 1, the most informative external comparison point is the best IWSLT24 English-to-German system's 68.18% feminine accuracy. That is a different language pair, so we cite it only as context.

Decisive checks. We run three. The first is whether listeners rate mismatched inflection as less natural at matched prosody, and whether TTSDS fails to register the drop. If listeners notice the mismatch only as a correctness error and not as unnaturalness, the naturalness framing collapses and the work becomes a fairness study. The second is whether output-conditioned decisions beat source-conditioned ones in the voice-changed regime while tying them in the unchanged regime. The third is whether the pitch-only ablation loses concordance on ambiguous voices relative to the full-timbre score, which would confirm that formant-level information must reach the decision.