Semantic Content Near the Origin: One Hyperbolic Latent for Two-Stage Speech Generation

Modern speech generators work in two stages. The first stage plans what is said as a short sequence of coarse "semantic" tokens. The second stage renders how it sounds: voice, prosody, room, and channel. To make this split, current systems either train two separate representations, such as a self-supervised content model plus an audio codec, or force a factorized codec apart with phoneme supervision and information bottlenecks. Either way, the two stages live in unrelated spaces. The second stage then has to learn, from scratch, which acoustic variations belong to which content. When it guesses wrong, the error shows up as mispronounced or smeared speech.

This note proposes building the split into the geometry of a single latent space. The space is hyperbolic, meaning curved so that available room grows exponentially as one moves away from the center, much like the number of leaves in a tree grows with depth. Each audio frame is encoded as one point. Its semantic token is read off by pulling the point back toward the origin, onto an inner shell, and quantizing it there. All acoustic detail lives in the outer region, inside the "cone" that fans out above that token. The acoustic stage then becomes a flow that carries a sample outward from its semantic anchor to a full acoustic point. Because cones widen exponentially, each content token gets a large private region for its many voices and conditions, and neighboring tokens' regions stay apart. The bet is that this one geometric constraint can replace the phoneme bottlenecks and dual-encoder designs that decoupling currently relies on. The main test case is generative speech enhancement, where the semantic plan must survive noise and the acoustic stage must restore a clean voice.

Core Claim

Speech has a natural one-to-many structure. One phonetic unit has many allophonic variants, each of those has many speaker realizations, and each realization has many recording conditions. A tree is the right shape for this. Hyperbolic space can hold trees with low distortion because ball volume grows exponentially with radius rather than polynomially. The claim is that matching the decoupling hierarchy to this geometry gives three measurable benefits.

First, semantic tokens read from the inner shell carry less speaker and noise information than Euclidean factorized tokens trained with the same invariance loss. Second, the acoustic stage confuses content less. We measure this as word error rate after resynthesis, because each anchor's cone has room for all its realizations without spilling into neighboring cones. Third, both benefits shrink smoothly as curvature goes to zero. That last property is what separates a geometric effect from an architectural accident. If a Euclidean "polar" version of the same design (direction for content, norm for detail) matches the hyperbolic one, the claim is false.

Where This Sits

The decoupling literature motivates the problem. GenSE chains a noisy-to-semantic language model on XLSR features (a frozen self-supervised speech model) with a semantic-to-speech model over a single-quantizer codec. The two spaces are unrelated, so timbre must be smuggled through prompt engineering. SISE factorizes a codec into semantic and acoustic tokens using an information bottleneck and supervised phoneme prediction, then runs semantic and acoustic diffusion in sequence. We keep SISE's two-stage generation but replace both its supervision and its two token streams with a radial split of one latent. SongBloom and Long-Form Speech Generation with Spoken Language Models show why the split matters at long horizons and how errors propagate from plan to render. We do not tackle long-form coherence here. ReGen reports that conditional flow matching for audio can collapse toward over-smoothed means and adds a Euclidean repulsive term. Our outward cone-restricted flow is a geometric alternative to that penalty.

Hyperbolic work supplies the tools. HyperVQ quantizes by hyperbolic multinomial logistic regression, a classifier whose decision boundaries are hyperbolic hyperplanes. On CIFAR-100 it raises codebook perplexity from 58.09 to 220.22. It is a single-stream quantizer, and we reuse it only for the inner shell. Hyperbolic Curvature as an Inductive Bias for Latent Space Flow Matching runs Riemannian flow matching on hyperbolic image latents. It is static, image-only, and unconditional. Path-Decoupled Hyperbolic Flow Matching places coarse prototypes at the origin and flows image features inward toward them for classification, and needs a stopping rule to avoid crowding at the origin. Our flow runs the opposite way: it is generative, starts at a coarse anchor, and moves outward into expanding volume, so origin crowding is not an issue.

Other papers are the closest neighbors and also constrain the design. HYFuse fuses frozen semantic and codec embeddings with Möbius addition (the hyperbolic analogue of vector addition) for emotion classification. It pools away time and never generates. Hyperbolic Audio Source Separation finds that distance from the origin tracks model certainty (ρ = 0.75 against Monte-Carlo dropout uncertainty), with ambiguous mixture bins sitting near the origin. That finding is a warning for us. Noisy frames might drift inward because they are uncertain, not because they are "semantic." We treat this as a confound to measure, not as support.

Research Program

The encoder maps 16 kHz audio to 50 Hz frame vectors. An exponential map at the origin places each frame vector as a point $z_t$ in the Lorentz model of hyperbolic space with curvature $-c$. We choose the Lorentz model to avoid the numerical problems at the boundary of the Poincaré ball. Let $v_t = \log_o(z_t)$ be the frame's tangent vector at the origin $o$, so that $\lVert v_t \rVert$ is its hyperbolic distance from the origin. The semantic point is the radial truncation of $z_t$ at a fixed inner shell of radius $r_s$:

$$s_t = \exp_o\!\left(\min\!\left(1, \tfrac{r_s}{\lVert v_t\rVert}\right) v_t\right).$$

A HyperVQ codebook on that shell assigns each $s_t$ a semantic token $k_t$ with code point $q_{k_t}$. Whatever $z_t$ holds beyond the shell, its residual angle within the code's cell and its radius, is acoustic information. That region widens exponentially with radius, so each token's acoustic capacity grows exponentially in $c$.

Nothing so far forces content to sit on the inner shell. That comes from the training objective. A decoder reconstructs the waveform from the full points $z_{1:T}$ using standard spectral and adversarial losses. An invariance term requires a content-preserving perturbed copy of the input to land in the same semantic cells. The perturbations are pitch and formant shifts, additive DNS-style noise, and reverberation. We write $s'_t$ for the semantic point of the perturbed copy and $\ell_k(\cdot)$ for HyperVQ's logit for code $k$:

$$\mathcal{L}_{\text{inv}} = -\sum_t \log \operatorname{softmax}_k\big(\ell_k(s'_t)\big)\Big|_{k=\mathrm{sg}(k_t)},$$

where $\mathrm{sg}$ stops gradients through the clean token. In words, the perturbed frame must classify into the clean frame's token, while the full point is free to differ. No phoneme labels are used.

The acoustic stage is a conditional Riemannian flow matching model over the frame sequence. It starts from a wrapped normal distribution (a Gaussian pushed onto the curved space) centered at each $q_{k_t}$. It transports samples outward along geodesics toward the target $z_t$, conditioned on the semantic token sequence and an utterance-level voice embedding. For enhancement, a small transformer predicts clean semantic tokens from noisy encoder points, playing the same role as GenSE's noisy-to-semantic model. The flow then renders clean acoustics, taking the voice embedding from the noisy input. We sweep curvature $c \in \{0, 0.5, 1, 3\}$, where $c = 0$ is the Euclidean polar limit, and the shell radius $r_s$. Encoder and decoder are retrained per setting, because the image work shows curvature cannot be changed after training.

Baselines

The naive reference point is the unprocessed noisy input for enhancement and plain codec resynthesis for factorization. It sets the floor on DNSMOS and word error rate and shows how much the semantic plan alone recovers.

The strongest existing enhancement system in the file is SISE. It reports a PMOS (overall quality) of 3.58 on the DNS 2021 test set, against 3.43 for MOSS-Former (SISE). It is the right competitor because it shares our structure, a factorized codec followed by two-stage generation, but enforces factorization with phoneme supervision in flat space. GenSE is the second reference: DNSMOS OVL of 3.43 on DNS without reverb, and 28.4% WER on CHiME-4 (GenSE). It represents the frozen self-supervised-semantics route.

Fairness rules for these comparisons are as follows. Where code is available, both systems are retrained on our training mixture with matched parameter count and step budget. We report their published numbers separately under their own protocols, and never mix PMOS with DNSMOS OVL. Beating SISE means at least matching its PMOS on DNS 2021 while using no phoneme labels, together with lower CHiME-4 WER than GenSE's 28.4%.

The monolithic baseline is a single-stream HyperVQ codec with no radial split, followed by the same flow model. It tests whether the split matters at all. No reported LibriSpeech number for HyperVQ was found on Lacuna, so we run it ourselves.

Two ablations isolate the key ingredient. The decisive one is the Euclidean polar twin at $c = 0$: identical encoder, decoder, invariance loss, codebook size, flow network, and hyperparameter grid, with content quantized by direction on a sphere of radius $r_s$. The second is a Euclidean two-branch factorized codec, with separate semantic and acoustic encoders, the same invariance loss, and no shared geometry. Every variant gets the same tuning budget, a fixed grid over learning rate and loss weights, and three seeds.

Why not just…?

Why not just use a Euclidean factorized codec with the same invariance loss? This is the real threat. The augmentation-invariance loss alone might do most of the decoupling. The two-branch and $c = 0$ polar ablations exist to answer this. The hyperbolic argument is specific: at a fixed radius budget, the acoustic region above each token is exponentially larger, so content confusion should be lower for the same codebook size. If the polar twin matches on leakage probes and resynthesis WER across all codebook sizes, we will report the loss, not the geometry, as the contribution.

Why not just take semantic tokens from a frozen self-supervised model, as GenSE does? Frozen models are strong and need no codec training. Their drawback is that the acoustic stage starts from an unrelated space, which is why GenSE needs token-chain prompting to keep timbre. GenSE is a retrained baseline, so if it matches us on CHiME-4 WER and speaker similarity, the shared-latent argument does not hold for enhancement.

Why not just fix codebook collapse with Euclidean tricks such as k-means initialization or GenSE's codebook reorganization? That would recover utilization, but utilization is not the claim. HyperVQ's perplexity gain is a side benefit. The prediction specific to our design is lower speaker information in tokens and lower content confusion in rendering. We report perplexity for every variant so a utilization gap cannot be mistaken for a geometry effect.

Why not just use residual vector quantization with a semantically distilled first layer? That design stacks many codebooks per frame, and GenSE argues this lengthens sequences for the planning model. We keep one semantic token per frame at 50 Hz, the token rate GenSE's single-quantizer codec targets. We do not run a full residual-quantization comparison, which limits how strongly we can claim superiority over that family.

Decisive Experiments

Factorization is measured on LibriSpeech 960h. We train linear probes on frozen semantic tokens for frame phone identity, using forced alignments, and for speaker identity, across the four curvatures and the ablations. Success means speaker probe accuracy falls with curvature while phone accuracy holds. Resynthesis quality is measured on LibriTTS at 24 kHz via ASR word error rate and speaker similarity. Here the hyperbolic model should beat the polar twin on WER at equal codebook size.

Enhancement is evaluated on DNS 2021 (SIG, BAK, OVL, and PMOS, the last under SISE's protocol) and on CHiME-4 word error rate, against all baselines.

Two diagnostics settle whether the geometry is doing what we say. The first is the certainty confound. We correlate each frame's radius with local signal-to-noise ratio. If noisy frames collapse inward and their semantic tokens degrade, the source-separation finding about uncertain points near the origin applies, and $r_s$ must move outward or be made input-dependent. The second is cone fidelity. For flow samples, we measure the fraction whose final point quantizes back to the conditioning token, as a function of curvature. A clear rise from $c = 0$ to $c = 1$, with no loss in DNSMOS, is the result that would justify the curvature. A flat curve would mean the radial split is sound engineering but the hyperbolic geometry adds nothing.