Expressive speech synthesis systems can now turn a dial for "how angry" or "how accented" a sentence sounds. They learn that dial from comparisons rather than ratings: the model is shown two recordings and told only which one has more of the attribute. This approach is called relative attribute ranking. Every current system applies the dial to the whole sentence at once, so it cannot place anger on one word or show a heavy accent on only the sounds a speaker actually struggles with. Collecting comparisons at the level of single words would be expensive and unreliable. We propose to learn per-phoneme intensity (a phoneme is a single speech sound) from the same cheap sentence-level comparisons that systems already use.
The mechanism has two parts. First, the transcript decides where to look: each phoneme in the text "asks" the audio for the frames that belong to it, through cross-modal attention, meaning attention in which one modality supplies the questions and another supplies the answers. Only the audio decides how intense each phoneme is. Second, every training pair has the identical sentence, which several standard corpora provide. Because the text is the same on both sides, it carries no information about which recording is more intense. This removes by construction the shortcut in which audio-language models read the words and ignore the voice. The sentence-level comparison then forces a set of per-phoneme intensity scores to emerge. We can check those scores against real ground truth: the L2-ARCTIC corpus contains human annotations of which phones a non-native speaker mispronounced.
Core Claim
Sentence-level pairwise comparisons between recordings of the same text are enough supervision to learn a phoneme-level intensity map. This holds provided two conditions are met. The score must be computed only from audio content routed by text-driven attention, and the sentence score must be a soft maximum over phonemes, so that "more intense overall" can be explained by a few phonemes becoming more intense. Such a map should localize real local events (mispronounced phones, spliced emotional words, question-final rises) without ever being trained on their locations. When fed into a synthesizer, it should allow per-word intensity editing that sentence-level systems cannot do.
In machine learning terms, this is ranking with multiple-instance supervision: labels attach to bags (sentences), and the model must infer which instances (phonemes) are responsible. The existing speech ranking literature has not used this formulation. Instead, it has either broadcast a single scalar or assumed local labels.
Where This Sits in Prior Work
The ranking side supplies the training signal. The idea of learning a scoring function from ordered pairs comes from Relative attributes. In speech it has been instantiated in several ways. Emotion Intensity and its Control for Emotional Voice Conversion treats neutral speech as zero intensity and emotional speech as higher. Fine-grained Emotional Control of Text-To-Speech blends neutral and emotional audio with a mixing weight and ranks on that weight. Controllable Accented Text-to-Speech Synthesis With Fine and Coarse-Grained Intensity Rendering (CAI-TTS) ranks non-native recordings against synthetic native references. We reuse the pair constructions from all three. However, each of these systems produces one number per utterance. CAI-TTS names the uniform sentence-level scalar as its main limitation, and that limitation motivates this proposal.
Two systems already go below the sentence level, and they constrain our design. MsEmoTTS ranks at the syllable level. To do so it must assign syllable-level orderings, which in practice means pushing the sentence-level ordering down uniformly onto every syllable. Our multiple-instance formulation avoids that assumption. QI-TTS hard-codes the final roughly 0.52 s as the region that carries questioning intonation. It reaches 95.2% intonation accuracy on ESD, but it cannot discover the location. We use its finding as a sanity check: our learned map should rediscover question-finality without being told. Attention-based Interactive Disentangling Network (AINN) learns a temporal saliency mask over audio frames with audio-only attention. It is the closest prior mechanism and the most important ablation, because it asks whether text-driven queries add anything over audio-only saliency.
The attention side supplies both a warning and a method. Do Audio LLMs Really LISTEN, or Just Transcribe? shows that when text and audio are fused, models lean on the words. Emotion accuracy on neutral-text emotional speech falls to 25–35%, and adding the neutral transcript makes it worse. An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention counters a similar dominance problem with a learned gate and sparse attention. We take a different route. Rather than learning a gate that hopefully suppresses the dominant modality, we make the dominant modality structurally unable to affect the score: text enters only through the queries, and the training pairs share their text. Among the papers in the file, EAD-VC comes closest to combining ranking and cross-attention. It uses them in separate stages, with ranking for pitch and rhythm on augmented audio and cross-attention only for injecting speaker timbre. Neither component localizes an attribute. Attribute-Guided Cross-Modal Interaction and Enhancement for Audio-Visual Matching reports that transformer cross-attention overfit and lost to CNNs on moderate data. That warning matters here because ESD and L2-ARCTIC are small. We therefore build on frozen pretrained speech features and keep the attention module light, following the frozen-expert design that CA²ST found effective.
The Model
Each phoneme $k$ of the transcript produces a query $q_k$ from a small text encoder. Frozen self-supervised speech features provide keys $K$ and values $v_t$ for each audio frame $t$. A light convolutional projection sits on top of those features to supply local inductive bias. The phoneme reads the audio as $c_k = \sum_t \alpha_{kt} v_t$, with $\alpha_{k\cdot} = \mathrm{softmax}(q_k K^\top/\sqrt{d})$, where $d$ is the key dimension. A monotonic prior keeps the attention near the forced alignment while letting it spill onto neighboring frames. The local intensity is $s_k = w^\top c_k$. It depends on the text only through which frames were read, never through the phoneme identity itself.
The sentence score is a soft maximum over phonemes:
$$S(x) = \tau \log \frac{1}{N}\sum_{k=1}^{N} \exp(s_k/\tau),$$
where $N$ is the number of phonemes and the temperature $\tau$ interpolates between mean pooling (large $\tau$) and max pooling (small $\tau$). For a pair $(x_i, x_j)$ with identical text in which $x_i$ is known to be more intense, we minimize $\max(0,\, m - S(x_i) + S(x_j))$ with margin $m$. A gradient-reversed speaker classifier on $c_k$, the same device QI-TTS uses against lexical leakage, discourages the score from encoding who is speaking. This matters most for accent, where the paired recordings come from different speakers. The temperature is the key design variable. A large $\tau$ reduces to the uniform assumption. A small $\tau$ says that a sentence is more intense if its most intense part is. We sweep $\tau$ and report localization as a function of it.
For synthesis, a FastSpeech 2 model receives the phoneme-level intensity sequence as an extra variance input. During training, the sequence comes from the frozen ranker; at inference, a text-based predictor or a user supplies it. We add a per-phoneme version of the CAI-TTS consistency loss: the ranker re-scores the generated audio, and its local scores must match the conditioning values. This prevents the generator from ignoring the input.
Research Program
The work proceeds on two attributes chosen because each has a localization ground truth that the model never sees. The first is emotion intensity on ESD (Seen and Unseen emotional style transfer for voice conversion with a new emotional speech dataset). ESD has 350 parallel sentences per speaker across five emotions, so every neutral–emotional and mixup pair can be text-matched. The second is accent strength on L2-ARCTIC. Its speakers read shared ARCTIC prompts, so non-native recordings can be paired against native references of the same sentence; where no native recording is used, we fall back to CAI-TTS-style synthetic native references. The 1,499 utterances with manually annotated substitutions, deletions and additions serve purely as a test set for localization.
For emotion we build the missing ground truth synthetically. We take a neutral and an emotional ESD recording of the same sentence by the same speaker, force-align both, and splice the emotional rendition of one word into the neutral utterance with short crossfades. Each spliced utterance therefore has a known intense region. A second set applies mixup only inside that word's span, which gives a known region and a known magnitude. These spliced items are held out of training.
Baselines
The naive reference point is a sentence-level ranker of the Emovox or CAI-TTS kind, trained on the same text-matched pairs, whose single score is broadcast to every phoneme. It defines chance localization and the controllability ceiling of current systems. The strongest existing ranking method is the mixup ranker of Fine-grained Emotional Control of Text-To-Speech. It reports average intensity controllability accuracy of 0.77 on EmoV-DB against 0.68 for FEC and 0.56 for RFTacotron, with MCD 4.66 dB and MOS 3.76. We retrain it on ESD with our pair sets and the same FastSpeech 2 backbone, so that sentence-level controllability is compared on equal footing. Beating it means matching its sentence-level controllability while adding word-level control it lacks; we do not claim to exceed it at the sentence level. For accent, the strongest sentence-level system is CAI-TTS, which reports MCD 6.08 dB and over 80% correlation between intended and perceived intensity on L2-ARCTIC. Our synthesizer should stay within noise of that MCD.
Two baselines target the localization claim directly. The first is a MsEmoTTS-style syllable ranker that inherits sentence orderings uniformly. The second is an AINN-style audio-only saliency mask whose mask-weighted frame scores are pooled per phoneme. As a non-learned reference for mispronunciation localization, the L2-ARCTIC paper reports Goodness of Pronunciation precision and recall of 0.29 for substitutions. That method uses phone-level acoustic models rather than intensity comparisons, so it is a reference point, not a like-for-like competitor. We found no reported result on Lacuna for unsupervised phone-level localization on L2-ARCTIC or for word-level emotion localization on ESD, so these comparisons are new.
The ablations isolate the key ingredient. The first replaces text-matched pairs with unmatched pairs of equal count. The second lets phoneme identity enter $s_k$ directly, a text-allowed score that recreates the lexical shortcut. The third replaces cross-attention with hard forced-alignment pooling. The fourth replaces the soft maximum with a mean. All variants share the frozen feature extractor, the parameter budget within 10%, the training steps, and an equal hyperparameter search of about 20 trials each, tuned on a validation split of spliced items disjoint from test.
Why not just…?
Why not just train a sentence-level ranker and attribute its score to frames with gradient saliency? This is the cheapest route to a local map, and it is a fair competitor. A ranker trained only to order whole sentences has no pressure to make its internal evidence local. Saliency maps over pooled scores are known to spread over energetic or voiced regions rather than over the contrastive ones. We run integrated-gradient attribution on the broadcast baseline and score it with the same localization metrics. If it matches the soft-max model, the contribution reduces to "use text-matched pairs," and we would report it that way.
Why not just rank at the syllable level as MsEmoTTS does? Syllable-level ranking needs syllable-level orderings, and those are not available. Pushing a sentence ordering onto every syllable assumes the uniformity we are trying to escape. On spliced ESD items this assumption predicts that the unspliced words in a spliced utterance rank as intense as the spliced word, which the localization test measures directly. If the ordering push-down matched our model anyway, it would mean real emotional speech is nearly uniform at this granularity. That would be a substantive negative finding about the task, not just about our method.
Why not just fuse text and audio with gated cross-attention to fight lexical dominance? The gated design of An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention learns to balance modalities. It offers no guarantee that the text pathway cannot carry the score, and LISTEN shows that the text pathway wins when it can. Our design removes the pathway rather than learning to down-weight it. The text-allowed-score ablation, with and without a gate, tests whether this matters. The test uses neutral-text emotional items in the style of LISTEN's neutral-text condition, built from ESD, which contains neutral sentences spoken emotionally. We expect the text-allowed variants to under-score emotional audio whenever the words are neutral.
Why not just use forced alignment and skip cross-attention? This is the most serious threat. Forced alignment on read speech is accurate, and hard pooling has fewer parameters, which matters given the overfitting reported by Attribute-Guided Cross-Modal Interaction and Enhancement for Audio-Visual Matching. The argument for soft attention is that pitch accents and accent-induced coarticulation spill across phone boundaries. The hard-pooling ablation decides the question. If hard pooling is within noise on both localization tests, we adopt it, and the proposal's claim narrows to multiple-instance ranking on text-matched pairs. That claim still stands without cross-modal attention.
Proposed Experiments and Decisive Checks
The first decisive check is accent localization on the annotated L2-ARCTIC subset. We compute the area under the ROC curve for detecting annotated mispronounced phones from $s_k$, along with precision and recall at the rate implied by the annotation density, for comparison against the 0.29 GOP figure. The model must clearly beat the broadcast baseline, which is at chance by construction, and the saliency baseline. A null result here, with real annotations rather than synthetic ones, would falsify the core claim for accent.
The second check is emotion localization on held-out spliced ESD items. We measure top-1 word accuracy (whether the highest-scoring word is the spliced one) and, on the region-limited mixup set, the Spearman correlation between local score and mixing weight. We report both across the $\tau$ sweep, to show the transition from uniform pooling to localization.
The third check reuses QI-TTS's finding on the roughly 310 questions per English ESD speaker. We train a questioning-intensity ranker with statements as the lower anchor and measure what fraction of local score mass falls in the final syllable. We also check whether the learned map ever puts mass elsewhere on question types that QI-TTS's fixed window would miss.
The fourth check is synthesis. We measure sentence-level controllability accuracy on ESD under the protocol of the mixup ranking paper, with MCD and MOS, and expect parity with the retrained mixup baseline. We then run a word-level test: listeners hear two syntheses of the same sentence in which intensity was raised on different words, and must identify which word was emphasized. Sentence-level systems cannot pass this test even in principle. Finally, the per-phoneme consistency loss is ablated to confirm that the generator actually uses the local input. The proposal succeeds if the first or second check clearly beats both the broadcast and saliency baselines, and if the synthesizer passes the word-level test at no cost to sentence-level controllability.
References
- Relative attributes
- Emotion Intensity and its Control for Emotional Voice Conversion
- Fine-Grained Emotional Control of Text-to-Speech: Learning to Rank Inter- and Intra-Class Emotion Intensities
- Controllable Accented Text-to-Speech Synthesis With Fine and Coarse-Grained Intensity Rendering
- MsEmoTTS: Multi-Scale Emotion Transfer, Prediction, and Control for Emotional Speech Synthesis
- QI-TTS: Questioning Intonation Control for Emotional Speech Synthesis
- Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversion
- Do Audio LLMs Really LISTEN, or Just Transcribe? Measuring Lexical vs. Acoustic Emotion Cues Reliance
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- EAD-VC: Enhancing Speech Auto-Disentanglement for Voice Conversion with IFUB Estimator and Joint Text-Guided Consistent Learning
- Attribute-Guided Cross-Modal Interaction and Enhancement for Audio-Visual Matching
- ${\text{CA}^{2}\text{ST}}$: Cross-Attention in Audio, Space, and Time for Holistic Video Recognition
- Seen and Unseen Emotional Style Transfer for Voice Conversion with A New Emotional Speech Dataset
- L2-ARCTIC: A Non-native English Speech Corpus