Abstract
Sequence modeling often suffers from a significant "research-to-production tax," where models optimized for parallel training must be manually rewritten for real-time streaming inference. This process is error-prone, leading to causality violations and implementation mismatches. We introduce SequenceLayers, a neural network API that enforces a strict contract between parallel and sequential execution. By utilizing a unified Sequence object and explicit state management, SequenceLayers ensures that training and inference code paths are identical, facilitating "streamable for free" model deployment.
Introduction
Building sequence models—such as those for speech recognition, translation, or large language models (LLMs)—is notoriously difficult due to the divergence between training and inference environments. During training, researchers typically use "teacher-forcing" to process entire sequences in parallel for computational efficiency. However, real-time deployment requires "streaming" inference, where data arrives incrementally.
This transition frequently introduces several critical issues:
- Causality Violations: Accidental "lookahead" during training that is impossible in real-time.
- Implementation Mismatch: The requirement to maintain two separate codebases (training vs. inference), which often diverge and introduce subtle bugs.
- State Management: The manual handling of complex internal states, such as Transformer KV caches or RNN hidden states.
- Padding and Masking: Ensuring that padding used for batching does not corrupt data during downsampling or pooling operations.
Methodology
SequenceLayers addresses these challenges by providing a unified API implemented in JAX and TensorFlow. The core of the framework is the Sequence object. Instead of passing raw tensors, layers consume and produce Sequence objects that bundle data values with a boolean mask. This ensures that masking information is preserved throughout the network, even during complex operations like downsampling.
Every layer in the framework must adhere to a strict contract by implementing three primary methods:
layer(): Processes a full sequence (optimized for training).get_initial_state(): Initializes internal states (e.g., buffers or caches).step(): Processes a single block of data and updates the state (optimized for streaming).
To ensure correctness, the library provides a verify_contract utility. This tool runs unit tests to confirm that processing a sequence in its entirety (layer) produces numerically identical results to processing it piece-by-piece (step).
As shown in Table 1, the framework explicitly tracks how different layer configurations affect the output ratio of the sequence. This metadata is crucial for maintaining synchronization across the network, especially when dealing with upsampling or downsampling layers. Complex models are constructed using declarative "combinators" such as Serial, Parallel, and Residual, which automatically manage the propagation of states and masks between sub-layers.
Applications and Use Cases
SequenceLayers is a production-hardened framework currently utilized across a wide array of applications at Google, including:
- Gemma 3n’s streaming audio encoder.
- Wave-Tacotron and Very Attentive Tacotron for Text-to-Speech.
- DolphinGemma and internal models for speech translation, vocoding, and video understanding.
Discussion
The framework provides several key advantages over traditional manual implementations:
- Explicit State Management: By making states (like KV caches) an explicit part of the API, debugging and deployment are simplified.
- Programmatic Latency Handling: The library tracks
input_latencyandoutput_latency, automatically managing the necessary padding or skipping required for layers that need future context (lookahead). - Production Readiness: Models can be exported as TensorFlow SavedModels and converted to LiteRT for mobile deployment without manual re-implementation of streaming logic.
While SequenceLayers significantly reduces the "research-to-production tax," it has some limitations. It is primarily designed for frameworks that compile to static computation graphs (XLA), making it less suitable for "dynamically clocked" pipelines where nodes produce output at irregular intervals. Additionally, while it handles end-padding robustly, managing start or interior padding for non-contiguous sequences remains an area for future development.
Conclusion
SequenceLayers offers a robust design philosophy for sequence modeling. By making the research code be the production code, it accelerates innovation and ensures the reliability of streaming neural networks in real-world applications.
Summary updated