Emotional text-to-speech systems try to make a chosen voice sound happy, angry or sad without changing whose voice it is. They fail in two familiar ways. In one, the voice drifts toward the speaker of the emotional reference clip. In the other, the output goes flat and neutral, because the easiest way to satisfy "keep the identity" and "express the emotion" together is to express nothing. Current fixes add local penalties inside the network. One is an adversary with gradient reversal that punishes speaker information in the emotion code. Another is a quantized bottleneck that throws away detail. A third is signal-processing scrambling of formants, the resonant peaks that carry a voice's timbre. Each penalty removes identity information by damaging the representation, and each shows the same trade-off between expressiveness and identity.
String theory solved a structurally similar problem long ago. When physicists compactify extra dimensions, they need to break a large gauge symmetry down to the Standard Model. Breaking it with local field strength (flux) distorts the geometry and ruins coupling unification. Wilson lines avoid this. A Wilson line is a flat gauge field, one with zero local field strength whose only effect is a phase picked up when going around a non-contractible loop. It breaks symmetry through global structure and leaves the local physics untouched. We propose to parameterize emotion in the same way. Emotion becomes a flat, commuting group action on a speaker's latent voice vector. Speaker identity is then an exact invariant by construction rather than something a penalty encourages. We pair this with a second borrowed tool, the analysis of tachyonic instability, meaning a direction with negative curvature at a symmetric point. We use it to explain and predict when a model collapses into the neutral output.
Core Claim
The claim has two parts. First, emotion is represented as a rotation by learned angles in a set of commuting two-dimensional planes of the speaker latent: a "holonomy," or transport around a loop, on a torus. Speaker identity is read only from quantities that the rotation leaves unchanged. This gives identity preservation without an adversary. It also keeps emotion's effect speaker-dependent, because rotating a speaker-specific vector changes each voice differently, and it removes identity drift when emotions are chained. Second, the neutral compromise is a loss of instability at the neutral point. Collapse happens when the curvature of the training loss along the emotion-intensity direction, measured at zero intensity, turns from negative to positive as the disentanglement weight grows. That sign flip can be measured early in training and should predict collapse before listening tests reveal it.
Concretely, let $z_s \in \mathbb{R}^{d}$ be speaker $s$'s neutral latent, split into $a_s \in \mathbb{R}^{d-2k}$ and $k$ two-dimensional blocks $b_{s,1},\dots,b_{s,k}$. Emotion $e$ with intensity $c \ge 0$ acts as
$$z_{s,e}(c) = \mathrm{diag}\big(I,\ R(c\,\theta_{e,1}),\ \dots,\ R(c\,\theta_{e,k})\big)\, z_s,$$
where $R(\phi)$ is the $2\times 2$ rotation by angle $\phi$ and $\theta_e \in \mathbb{R}^k$ are learned angles shared across speakers. The rotations commute, so the "connection" is flat: any sequence of emotion edits composes into the rotation by the summed angles, and returning to neutral returns exactly to $z_s$. The speaker encoder and any speaker classifier see only the invariants $(a_s, \lVert b_{s,1}\rVert, \dots, \lVert b_{s,k}\rVert)$. The decoder sees the full rotated vector, so emotion can still move formant-bearing dimensions. This matters because DiEmo-TTS found that intense emotion genuinely shifts timbre.
For the diagnostic, write the training loss as $L(c;\lambda)$, where $\lambda$ is the weight on the disentanglement term. The adversarial weight for gradient-reversal baselines or the invariance weight for ours plays the role that curvature coupling $\xi$ plays in $m^2_{\text{eff}} = m_0^2 + \xi R$. We track
$$m^2(\lambda) = \left.\frac{\partial^2 L}{\partial c^2}\right|_{c=0},$$
computed by Hessian-vector products. The physics lens predicts a critical $\lambda^\ast$ with $m^2(\lambda^\ast)=0$ beyond which emotional intensity stops growing. The first derivative at $c=0$ need not vanish for real speech objectives, so we record it alongside the curvature and report whether curvature alone, or the pair, predicts collapse.
Relation to Prior Work
Four speech papers motivate the gap by showing the identity–expressiveness trade-off under local penalties. DiEmo-TTS perturbs formants and distills with DINO, a self-supervised teacher–student method. It reports beating gradient-reversal and quantized-bottleneck transformer baselines on speaker similarity. AgentSteerTTS names the "neutral compromise" and needs an agent loop driven by a multimodal large language model plus a 100-hour prototype library to escape it. iEmoTTS routes emotion through explicit pitch, energy and duration predictors, with a quantized timbre encoder. Emo-StarGAN adds emotion-preserving losses to voice conversion but still scores 0% on "Surprise" in RAVDESS.
The closest existing idea is DiEmo-TTS's relative emotion mapping. It defines each speaker's neutral center with a convex hull in a valence–arousal–dominance space and expresses emotion as a shift from it. We share the "emotion relative to a speaker's own neutral" intuition, but DiEmo-TTS's shift is additive and unconstrained, so it can move identity. Ours is a group action whose invariants carry identity exactly.
The method supply comes from Gauge Equivariant Transformer. It builds parallel transport with cyclic-group features into attention on 3D meshes, reporting 99.2% SHREC accuracy with 11K parameters. We reuse its practice of carrying group-valued features through attention, but the group acts on a voice latent rather than on tangent frames.
The physics papers constrain and inspire rather than supply code. A Z2×Z2 standard model shows that breaking symmetry with a freely acting discrete Wilson line avoids the damage local breaking does to unification. Heterotic Z6-II MSSM orbifolds in blowup shows the opposite case: smoothing by local blowup makes hypercharge anomalous. That is the "local breaking damages what you wanted to keep" pattern we attribute to gradient reversal. Calabi-Yau compactifications of non-supersymmetric heterotic string theory is the one paper in the file joining Wilson lines and tachyons. It uses discrete Wilson lines for symmetry breaking in a background chosen so tree-level tachyons disappear, and proves a minimum Wilson line order (at least four for SO(10) to the Standard Model). That result motivates our ablation on how many independent rotation planes are needed to separate five emotions. Hell's analysis of non-minimally coupled Proca fields supplies the template for the diagnostic: a coupling constant tunes an effective mass through zero and changes which modes are unstable.
Research Program
We build the model on a FastSpeech 2 acoustic backbone, the same family DiEmo-TTS uses, so conditioning differences are isolated. A reference encoder predicts the emotion angles $\theta_e$ and a scalar intensity $c$ from an emotional clip. A speaker encoder outputs $z_s$, and speaker supervision touches only the rotation invariants. Because identity is invariant by construction, no gradient reversal is used anywhere. The only extra loss is emotion classification on the angles.
We analyze three things. The first is the minimum torus dimension $k$ needed to separate ESD's five emotions and "Surprise" in particular, echoing the Wilson-line-order result. The second is whether learned angles show a continuous intensity axis like the arousal gradient emotion2vec observed. The third is the curvature diagnostic $m^2(\lambda)$ swept across $\lambda$ for both our model and the gradient-reversal baseline.
Baselines
The naive reference point is the same FastSpeech 2 backbone with concatenated speaker and emotion embeddings and no disentanglement term. It sets the floor for identity leakage and the ceiling for raw expressiveness. The classic local-penalty baselines are Trans-GRL, which uses gradient reversal, and Trans-VQ, which uses a quantized bottleneck, as benchmarked in DiEmo-TTS. They matter because they are exactly the mechanisms our argument says are harmful. We sweep their penalty weight so they are not strawmen.
The strongest reproducible competitor is DiEmo-TTS itself. It targets the same cross-speaker transfer task on ESD and already contains a relative-to-neutral emotion geometry. For it, the file reports naturalness MOS 3.80 on the Surprise category and WER 16.16%. The file reports no decimal speaker-similarity numbers for DiEmo-TTS or its baselines, so we must re-measure all of them ourselves. AgentSteerTTS reports emotion similarity 0.955 (against 0.864 for IndexTTS2) and speaker similarity 0.841 on its composite-instruction benchmark. It depends on a closed-loop large-language-model critic and a curated library, and it evaluates a different task, so we cite these numbers as context rather than as a head-to-head target.
Every trained system uses the same ESD split, the same backbone size within 5% of parameters, the same vocoder, and the same hyperparameter search budget. That budget is a fixed number of runs over the disentanglement weight and learning rate.
Three ablations isolate the key ingredient. In the first, the same rotation architecture has its generators replaced by unconstrained learned matrices, so emotion edits no longer commute: a "curved connection." In the second, the rotation is replaced by an additive speaker-relative offset with an identity-invariance penalty, a learned version of DiEmo-TTS's relative mapping. In the third, the rotation is kept but speaker supervision is applied to the full vector rather than to the invariants.
Beating DiEmo-TTS means better speaker similarity on unseen held-out speakers with emotion reclassification accuracy no worse, or better accuracy at matched similarity, with confidence intervals excluding zero across three seeds.
Why not just…?
Why not just split the latent into a speaker block and an emotion block with an orthogonality penalty? This is the most serious threat, because it is simpler and also gives identity invariance for free. The difference is that a disjoint emotion block has a speaker-independent effect on the decoder input. Our rotation of a speaker-specific vector produces a speaker-dependent change in the dimensions that carry timbre. The DiEmo-TTS evidence that intense emotion shifts formants suggests that dependence is needed. The concatenation baseline and the additive-offset ablation test this directly. If a split latent matches us on Surprise and on zero-shot speakers, the geometric construction is unnecessary and we will say so.
Why not just tune gradient reversal better, or add a margin loss pushing emotion codes away from neutral? A margin or double-well penalty on intensity is exactly what a "tachyonic potential" would be, so we do not claim such a term as new. We use the instability idea only as a diagnostic. The gradient-reversal sweep settles whether better tuning closes the gap. If a tuned Trans-GRL with a margin matches our identity scores without collapse, the local-penalty story is wrong. The file's evidence points the other way: DiEmo-TTS beat Trans-GRL, and AgentSteerTTS reports neutral compromise under adversarial training.
Why not just control pitch, energy and duration explicitly? Empathic Machines and iEmoTTS show such levers work and leave the spectral envelope untouched. By the same token they cannot express the spectral changes that come with high arousal. We include an iEmoTTS-style prosody-only variant and expect it to lose specifically on Angry and Surprise emotion reclassification.
Why not just use a strong self-supervised emotion encoder such as emotion2vec as the reference encoder? The emotion2vec authors themselves list speaker–emotion disentanglement as unsolved. It is also complementary rather than competing, so we test it as a drop-in reference encoder for both our model and DiEmo-TTS.
Proposed Experiments
We train on ESD's 10 English speakers and hold out 2 for zero-shot evaluation, repeating with the Chinese half. ESD's parallel utterances make loop tests possible. RAVDESS serves as an out-of-domain emotion transfer test, where Emo-StarGAN's 0% Surprise result sets a clear failure point. Objective metrics are speaker-embedding cosine similarity from an external verification model, WER from an external recognizer, and emotion reclassification accuracy from a linear probe on frozen emotion2vec. For the probe we cite as reference the reported 71.79% weighted accuracy on IEMOCAP and the ESD database paper's 92% (Chinese) and 89% (English) recognition accuracy. Naturalness and speaker-similarity MOS come from listening tests under the same protocol for every system.
Three checks decide the proposal. The first is a loop-closure test. We chain neutral → happy → angry → sad → neutral transfers, resynthesizing between steps with each system's own encoder, and measure speaker-similarity drift from the original. The flat construction predicts near-zero drift. The curved-connection ablation and DiEmo-TTS should drift, and if they do not, flatness buys nothing. The second is the Surprise test: whether any torus dimension $k$ separates Surprise from Anger and Happy in reclassification, and the smallest $k$ that does. The third is the collapse-prediction test. For each $\lambda$ in the sweep, we compute $m^2(\lambda)$ in the first 5% of training and ask whether its sign predicts the final emotion reclassification accuracy dropping below the naive baseline's. The prediction succeeds if rank correlation across the sweep is high for both our model and Trans-GRL. It fails, and the instability lens is dropped, if early curvature carries no information beyond the penalty weight itself.
References
- DiEmo-TTS: Disentangled Emotion Representations via Self-Supervised Distillation for Cross-Speaker Emotion Transfer in Text-to-Speech
- AgentSteerTTS: A Multi-Agent Closed-Loop Framework for Composite-Instruction Text-to-Speech
- iEmoTTS: Toward Robust Cross-Speaker Emotion Transfer and Control for Speech Synthesis Based on Disentanglement Between Prosody and Timbre
- Emo-StarGAN: A Semi-Supervised Any-to-Many Non-Parallel Emotion-Preserving Voice Conversion
- Gauge Equivariant Transformer
- A <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" altimg="si1.gif" overflow="scroll"><mml:msub><mml:mi mathvariant="double-struck">Z</mml:mi><mml:mn>2</mml:mn></mml:msub><mml:mo>×</mml:mo><mml:msub><mml:mi mathvariant="double-struck">Z</mml:mi><mml:mn>2</mml:mn></mml:msub></mml:math> standard model
- Heterotic ℤ6–IIMSSM orbifolds in blowup
- Calabi-Yau compactifications of non-supersymmetric heterotic string theory
- Unveiling the Inconsistency of the Proca Theory with Nonminimal Coupling to Gravity
- emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation
- Empathic Machines: Using Intermediate Features as Levers to Emulate Emotions in Text-To-Speech Systems