Dynamic Chunk Training for Flexible Latency Speech Models

16 concepts12 papers1 proposal

In speech and audio processing, developers have traditionally faced a rigid architectural trade-off. If an application requires immediate, real-time outputs—such as live captioning or simultaneous translation—the model must be trained as a streaming system, processing audio in short, fixed windows (or chunks) with minimal look-ahead context. If the application prioritizes maximum accuracy over speed—such as offline transcription—the model is trained as a non-streaming system, consuming the entire utterance at once to leverage global context.

Historically, satisfying both use cases meant training, deploying, and maintaining two entirely separate neural networks. Dynamic Chunk Training (DCT)—also referred to as variable-latency training—fundamentally changes this paradigm. Instead of fixing the temporal window size during training, DCT exposes the model to audio segments of randomly varying lengths. This simple shift forces the network to learn elastic representations, enabling a single deployed model to smoothly trade accuracy for latency at runtime without requiring retraining.


Breaking the Fixed-Window Constraint

To understand why traditional models fail when context lengths change, consider how self-attention and convolutional layers process sequence data. When a model is trained exclusively on full-length utterances, its attention mechanisms learn to rely heavily on long-range future context to disambiguate acoustic signals. If you suddenly force this model to operate on a 200-millisecond streaming chunk at inference time, its performance collapses because the expected future context is missing. Conversely, a model trained strictly on fixed 200ms chunks cannot exploit the richer global context available during offline processing.

DCT resolves this by treating the processing window size as a training hyperparameter that changes with every batch. As pioneered by Yao et al. in the WeNet toolkit, the training process randomly alternates between short audio chunks, medium segments, and full-length utterances.

By forcing the network to generalize across different temporal resolutions, the model learns to adapt its internal representations dynamically. When processing a short chunk, it relies on immediate local features; when given a larger window, it automatically extends its attention span. Raj et al. demonstrated the robustness of this approach in Continuous Streaming Multi-Talker ASR, showing that chunk width randomization allows a single model to handle highly complex, overlapping multi-speaker scenarios across a wide spectrum of latency targets.


Context Inheritance and Look-Ahead Windows

Varying the chunk size solves the problem of flexibility, but processing audio in isolated, short blocks introduces a new challenge: the loss of historical and future context. If a model only sees a isolated 100ms window, it loses the broader phonetic and semantic trajectory of the sentence.

To maintain high accuracy during low-latency operations, modern DCT frameworks incorporate two complementary mechanisms:

  1. Context Inheritance and Carry-Over: Instead of treating each audio chunk as an independent sequence, models pass historical state information forward. Deng et al. utilized block-based audio processing with context inheritance in their Blockwise Streaming Transformer, allowing the network to naturally carry historical context across boundaries rather than waiting for fixed token delays. To further optimize this, Huybrechts et al. introduced a dynamic context carry-over mechanism in the DCTX-Conformer, which passes a summarized history vector between chunks, yielding a 25% relative improvement in Word Error Rate (WER) with virtually no added latency.
  2. Elastic Look-Ahead Windows: While history can be carried forward indefinitely, looking into the future inherently adds latency. To manage this, architectures like the Flexi-Transducer developed by Mahadeokar et al. dynamically adjust both the processing chunk size and the future "look-ahead" window during inference. This allows a single model to instantly pivot from a zero-look-ahead mode for rapid voice commands to an extended-look-ahead mode for high-accuracy dictation.

Unified Supernets and Pre-training

Because a model trained with DCT can operate at any point along the latency-accuracy spectrum, researchers have used it to build "unified" or "dual-mode" systems. These systems embed multiple operational profiles inside a single set of weights.

For example, Liu et al. developed a dual-mode framework in Learning a Dual-Mode Speech Recognition Model via Self-Pruning. By combining dynamic segment sampling with self-pruning during training, they created a single "supernet" that can be deployed as a compact, sparse streaming model on edge devices, or as a large, dense non-streaming model on servers. Similarly, Sudo et al. leveraged dynamic block training in Time-synchronous one-pass Beam Search to run online and offline transducer decoders in parallel, combining their search spaces to improve decoding robustness.

Traditional Training (Fixed Context):
Streaming Mode:     [Chunk 1] -> [Chunk 2] -> [Chunk 3]  (Low Latency, Low Accuracy)
Offline Mode:       [               Full Utterance             ]  (High Latency, High Accuracy)

Dynamic Chunk Training (Variable Context):
Batch 1 (Short):    [Chunk 1] -> [Chunk 2] -> [Chunk 3]
Batch 2 (Medium):   [   Medium Chunk 1   ] -> [   Medium Chunk 2   ]
Batch 3 (Full):     [               Full Utterance             ]
Result: A single set of weights optimized for any runtime chunk size.

This flexibility is increasingly being baked into foundational models during the self-supervised pre-training phase. In GigaAM, Kutsakov et al. integrated a dynamic chunking strategy directly into self-supervised learning. This ensures that the underlying speech representations are inherently robust to chunk boundaries before any downstream fine-tuning occurs. To make these variable-context operations computationally efficient, van Dalen et al. introduced Linear Time Complexity Conformers, pairing dynamic chunk training with specialized Dynamic Chunk Convolution (DCCONV) layers to maintain linear time and memory scaling regardless of the chosen window size.


Generalization Across the Speech Pipeline

While initially designed for Automatic Speech Recognition (ASR), the core intuition of dynamic chunking—that varying temporal resolution during training breeds runtime operational elasticity—has spread across the entire speech and language processing pipeline.

  • Voice Conversion: In DualVC 3, Wang et al. applied variable chunk sizes to streaming voice conversion. This strategy allows the low-latency streaming path of the model to inherit structural, global knowledge from the full-utterance processing path, stabilizing real-time voice transformations.
  • Speech Translation: Zhang et al. developed StreamSpeech, an "all-in-one" simultaneous speech-to-speech translation model. By utilizing variable input chunk sizes during multi-task learning, a single system can seamlessly adapt to diverse real-time communication scenarios and latency constraints.
  • Downstream Text Normalization: The need for variable-latency processing extends to text-based modules that consume speech outputs. Ho et al. implemented a dynamic chunk-aware approach for Inverse Text Normalization (ITN). By dynamically adjusting processing chunk sizes during training and inference, the ITN language model can natively align with the variable, streaming output lengths generated by upstream ASR systems.

By shifting the training paradigm from static temporal windows to dynamic, variable-length chunks, these systems eliminate the engineering friction between research and deployment. A single model can be trained once and deployed everywhere, adapting its latency and accuracy on the fly to match the constraints of the hardware and the needs of the user.

Key Foundational Work

WeNet: Production Oriented Streaming and Non-Streaming End-to-End Speech Recognition Toolkit (2021) — contributed 2 concept elements to this direction.
Flexi-Transducer: Optimizing Latency, Accuracy and Compute for Multi-Domain On-Device Scenarios (2021) — contributed 2 concept elements to this direction.
Continuous Streaming Multi-Talker ASR with Dual-path Transducers (2021) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_variable_latency_training_99256927,
  title = {variable-latency training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variable-latency-training/rdr_992569278e914544919d967a647e878f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_variable_latency_training_99256927_v1,
  title = {variable-latency training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variable-latency-training/rdr_992569278e914544919d967a647e878f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake