Stream-Routed Low-Rank Adapters for Hybrid Discrete–Continuous Speech Input

Speech-understanding systems built on large language models usually hear audio in one of two forms. The first is discrete tokens: audio clustered into a small vocabulary of symbols, which is compact and captures phonetic identity well. The second is continuous features: the raw vectors of a pretrained speech encoder, which keep emotion, speaker, and noise detail. A careful comparison found that neither form wins everywhere. Continuous features win five of six understanding tasks, and win emotion recognition by nearly thirty points. Discrete tokens win phoneme recognition and converge about three times faster (Speech Discrete Tokens or Continuous Features?). The natural next step is to feed both forms at once. The authors leave that step as future work.

We argue that the step will quietly fail under the standard cheap fine-tuning method. That method is low-rank adaptation (LoRA): the large model is frozen, and each weight matrix learns only a small additive correction built from two thin matrices. When one shared correction serves both input streams, we expect the fast-converging discrete stream to dominate it. The model then learns to answer from the tokens and largely ignores the continuous features it was given. We propose stream-routed LoRA, in which the correction applied at each sequence position depends on whether that position holds a discrete token or a continuous frame. Each stream then gets its own adapter and its own learning rate. The claim is testable. A hybrid model adapted this way should match or beat the better pure pipeline on every task, whereas a hybrid model with one shared adapter should behave measurably like a discrete-only model.

Core Claim

Consider a hybrid input sequence built by concatenating the deduplicated, subword-merged discrete tokens of an utterance with its projected continuous frames. Each position $t$ carries a stream label $m(t) \in \{\text{disc}, \text{cont}\}$. In every adapted projection of the frozen language model we replace the shared update with

$$h_t = W_0 x_t + B_{m(t)} A_{m(t)} x_t,$$

Here $W_0$ is the frozen weight, $x_t$ is the input at position $t$, and $B_{m}A_{m}$ is a rank-$r_m$ correction owned by stream $m$. Attention still mixes information across streams, because the keys and values at each position come from its own stream's adapter while the queries can attend anywhere. Only the parameter update is separated.

The central hypothesis is mechanistic. Discrete positions reach low loss in 4–5 epochs, against 10–15 for continuous ones (Speech Discrete Tokens or Continuous Features?). A shared low-rank subspace is small, so the discrete stream's early gradients fill it. After that, the continuous stream has little room left to shape the adapter. Routing removes the competition. Per-stream learning rates, which a shared adapter cannot offer, address LoRA's known sensitivity to learning rate (A Unified Study of LoRA Variants) directly on the axis where the two streams differ most.

Relation to Prior Work

Three papers motivate the gap.

  • The discrete-versus-continuous comparison (Speech Discrete Tokens or Continuous Features?) supplies the complementary-strengths evidence, the convergence gap, and the finding that discrete tokens align with the model's internal text representations around layer 22 versus layer 28 for continuous features. It treats the two forms as mutually exclusive and does not study low-rank adaptation. We combine the two forms and study exactly that adaptation.
  • LoRA-Whisper (LoRA-Whisper) shows that separate LoRA modules per language stop languages interfering in a frozen recognizer, lowering average WER from 11.68% to 9.52%. It routes by language at the utterance level and supplies our method. We route by input stream at the position level, within a single utterance.
  • The speech PEFT study (PEFT for Speech) finds that ensembles of diverse adapters beat single adapters on SUPERB (2239 vs. 2140). It also found that automated per-layer placement did not beat uniform placement. That negative result constrains us: we use uniform placement and do not add a depth search. All of its adapters see one continuous representation, not two heterogeneous ones.

Two lines of work show that combining discrete and continuous signals pays off, but none adapts a frozen language model with low-rank updates.

  • Generative hybrids. The variational spoken language model (A Variational Framework for Improving Naturalness) adds continuous paralinguistic latents to discrete tokens and raises naturalness MOS from 3.19 to 3.60. Mel-QCD (Synchronized Video-to-Audio Generation via Mel-QCD) splits spectrograms into quantized and continuous parts. Both are trained from scratch or fully trained.
  • Continuous-in, discrete-out pipelines. LauraGPT (LAURAGPT) and LLaSE-G1 (LLaSE-G1) use continuous features as input and discrete tokens as output. They assign each form to a different side of the model, never both to the input, and both use full-parameter training.

Our setting sits in the gap these papers leave: both streams on the input side of a frozen language model, adapted with low-rank updates.

Research Program

Setup. We reproduce the controlled setup of the comparative study. The speech encoder is WavLM-Large. The discrete stream uses k-means with 2,000 clusters, followed by deduplication and byte-pair merging. The continuous stream uses a linear projection with frame concatenation. We use two backbones, Qwen1.5-0.5B and Llama3.1-8B, and the study's six tasks: speech recognition, phoneme recognition, keyword spotting, emotion recognition, intent classification, and speech translation. We add CHiME-4 for noisy speech recognition. The hybrid input is the discrete sequence, a separator, and then the continuous sequence. Concatenation sidesteps the alignment problem between deduplicated tokens and frames, so we can study adaptation without also inventing a new fusion scheme.

Adapter placement and capacity. Adapters sit uniformly on the attention and feed-forward projections. Routed and shared variants are matched on trainable parameters: the routed model uses ranks $r_{\text{disc}} + r_{\text{cont}} = r$, and the shared model uses rank $r$. Within that budget we also test unequal splits. This asks, empirically and cheaply, whether the discrete stream needs less rank than the continuous one.

Diagnostics. Beyond task scores, we measure how much each trained model relies on each stream. We zero out one stream at test time and record the accuracy drop. We also compute the gradient-norm share each stream contributes to a shared adapter over training. As a secondary question, we repeat the comparative study's layer-wise text-alignment probe. This checks whether routing moves the continuous stream's alignment layer earlier than 28.

Baselines

Pure discrete pipeline with vanilla LoRA. This is the reference that shows what the hybrid must beat on phoneme recognition and speed. It uses the same backbone, encoder, and data, with LoRA of rank $r$ and a learning-rate sweep of the same size as every other arm. On this setup, no reported result for phoneme recognition error was found on Lacuna. The comparative study states only that discrete tokens have a "clear advantage", so we reproduce the number ourselves.

Pure continuous pipeline with vanilla LoRA. This is the strongest competitor, because continuous features already win five of six tasks. The published anchor is emotion recognition accuracy of 65.87% for WavLM continuous features on Llama3.1-8B (Speech Discrete Tokens or Continuous Features?). That figure was obtained with a trained bridge rather than LoRA, so we rerun it under our LoRA budget. We report both the published and the reproduced value. Beating it means matching its emotion accuracy within noise across three seeds while also beating it on phoneme recognition and CHiME-4 WER. No reported CHiME-4 number for this setup was found on Lacuna.

Hybrid input with one shared LoRA, learning rate tuned. This is the ablation that isolates routing. Following A Unified Study of LoRA Variants, it gets a learning-rate sweep equal in size to the routed model's joint sweep over two learning rates. Otherwise, the routed model's extra tuning freedom would be an unfair advantage.

Routed LoRA with a single learning rate. This separates the effect of dedicated subspaces from the effect of per-stream step sizes.

Shared LoRA trained with stream dropout. Here the discrete stream is randomly removed during training. This is the cheapest direct fix for the shortcut we hypothesize.

Fairness. All arms share data, epochs, trainable-parameter count, tuning budget, and three seeds.

Why not just…?

Why not just use continuous features alone with vanilla LoRA? This is the most serious threat. The comparative study shows continuous features winning most tasks, and a hybrid input roughly doubles sequence length, which erases the discrete stream's compression advantage at inference. The hybrid is only justified if it wins on the union of tasks. That means recovering the discrete pipeline's phoneme-recognition advantage and improving noise robustness without losing emotion accuracy. If routed hybrid equals continuous-only everywhere, we will report that feeding discrete tokens alongside continuous features adds nothing under LoRA, which is itself a useful negative result for the future-work direction the comparative study proposed.

Why not just use one shared LoRA on the hybrid input and tune the learning rate carefully? The unified study found that tuned vanilla LoRA matches most variants, so this is a real alternative. Our hypothesis predicts a specific failure: with the continuous stream zeroed at test time, the shared model's emotion accuracy should drop only a little, because it learned to rely on tokens. The routed model should drop sharply. If the shared model instead relies on both streams and matches routed accuracy, our mechanism is wrong and we will say so.

Why not just ensemble a discrete-LoRA system and a continuous-LoRA system, as PEFT for Speech did with diverse adapters? Ensembling doubles inference cost. It can only combine outputs, so it cannot let continuous evidence correct a token that noise flipped mid-utterance. We run a two-system logit-average ensemble at matched total parameters. If it matches the routed model, the argument for in-model fusion weakens to a cost argument alone.

Why not just use stream dropout instead of routing? Dropout forces the model to use continuous features, but it also hurts the discrete stream's contribution on tasks like phoneme recognition. We include it as a baseline. Routing is preferred only if it beats dropout on phoneme recognition at equal emotion accuracy.

Decisive Checks

The proposal succeeds if three conditions hold on both backbones.

  • Union of strengths. Routed hybrid LoRA matches the reproduced continuous-only emotion accuracy (the published figure is 65.87%) and matches or beats the discrete-only phoneme error.
  • Mechanism. The stream-ablation test shows the shared-adapter hybrid relying mostly on the discrete stream, while the routed hybrid relies substantially on both.
  • Noise robustness. On CHiME-4, routed hybrid WER falls below discrete-only WER, showing that continuous positions rescue tokens flipped by noise.

The proposal fails cleanly in either of two cases: tuned shared LoRA shows balanced stream reliance and matches routed accuracy, or continuous-only LoRA matches the routed hybrid on every task. Either outcome answers the open question from the comparative study, at a cost of a few LoRA runs per task.