SequenceLayers: Sequence Processing and Streaming Neural Networks Made Easy

R. J. Skerry-Ryan, Julian Salazar, Soroosh Mariooryad, David Kao, Daisy Stanton, Eric Battenberg, Matt Shannon, Ron J. Weiss, Robin Scheibler, Jonas Rothfuss, Tom BagbyarXiv 2025 · DOI · arXiv

Abstract

Sequence modeling often suffers from a significant "research-to-production tax," where models optimized for parallel training must be manually rewritten for real-time streaming inference. This process is error-prone, leading to causality violations and implementation mismatches. We introduce SequenceLayers, a neural network API that enforces a strict contract between parallel and sequential execution. By utilizing a unified Sequence object and explicit state management, SequenceLayers ensures that training and inference code paths are identical, facilitating "streamable for free" model deployment.

Introduction

Building sequence models—such as those for speech recognition, translation, or large language models (LLMs)—is notoriously difficult due to the divergence between training and inference environments. During training, researchers typically use "teacher-forcing" to process entire sequences in parallel for computational efficiency. However, real-time deployment requires "streaming" inference, where data arrives incrementally.

This transition frequently introduces several critical issues:

  1. Causality Violations: Accidental "lookahead" during training that is impossible in real-time.
  2. Implementation Mismatch: The requirement to maintain two separate codebases (training vs. inference), which often diverge and introduce subtle bugs.
  3. State Management: The manual handling of complex internal states, such as Transformer KV caches or RNN hidden states.
  4. Padding and Masking: Ensuring that padding used for batching does not corrupt data during downsampling or pooling operations.

Methodology

SequenceLayers addresses these challenges by providing a unified API implemented in JAX and TensorFlow. The core of the framework is the Sequence object. Instead of passing raw tensors, layers consume and produce Sequence objects that bundle data values with a boolean mask. This ensures that masking information is preserved throughout the network, even during complex operations like downsampling.

Every layer in the framework must adhere to a strict contract by implementing three primary methods:

  • layer(): Processes a full sequence (optimized for training).
  • get_initial_state(): Initializes internal states (e.g., buffers or caches).
  • step(): Processes a single block of data and updates the state (optimized for streaming).

To ensure correctness, the library provides a verify_contract utility. This tool runs unit tests to confirm that processing a sequence in its entirety (layer) produces numerically identical results to processing it piece-by-piece (step).

Table 1: Example layer configurations and their corresponding output ratios within the SequenceLayers framework.

As shown in Table 1, the framework explicitly tracks how different layer configurations affect the output ratio of the sequence. This metadata is crucial for maintaining synchronization across the network, especially when dealing with upsampling or downsampling layers. Complex models are constructed using declarative "combinators" such as Serial, Parallel, and Residual, which automatically manage the propagation of states and masks between sub-layers.

Applications and Use Cases

SequenceLayers is a production-hardened framework currently utilized across a wide array of applications at Google, including:

  • Gemma 3n’s streaming audio encoder.
  • Wave-Tacotron and Very Attentive Tacotron for Text-to-Speech.
  • DolphinGemma and internal models for speech translation, vocoding, and video understanding.

Discussion

The framework provides several key advantages over traditional manual implementations:

  • Explicit State Management: By making states (like KV caches) an explicit part of the API, debugging and deployment are simplified.
  • Programmatic Latency Handling: The library tracks input_latency and output_latency, automatically managing the necessary padding or skipping required for layers that need future context (lookahead).
  • Production Readiness: Models can be exported as TensorFlow SavedModels and converted to LiteRT for mobile deployment without manual re-implementation of streaming logic.

While SequenceLayers significantly reduces the "research-to-production tax," it has some limitations. It is primarily designed for frameworks that compile to static computation graphs (XLA), making it less suitable for "dynamically clocked" pipelines where nodes produce output at irregular intervals. Additionally, while it handles end-padding robustly, managing start or interior padding for non-contiguous sequences remains an area for future development.

Conclusion

SequenceLayers offers a robust design philosophy for sequence modeling. By making the research code be the production code, it accelerates innovation and ensures the reliability of streaming neural networks in real-world applications.

Summary updated

Abstract

We introduce a neural network layer API and library for sequence modeling, designed for easy creation of sequence models that can be executed both layer-by-layer (e.g., teacher-forced training) and step-by-step (e.g., autoregressive sampling). To achieve this, layers define an explicit representation of their state over time (e.g., a Transformer KV cache, a convolution buffer, an RNN hidden state), and a step method that evolves that state, tested to give identical results to a stateless layer-wise invocation. This and other aspects of the SequenceLayers contract enables complex models to be immediately streamable, mitigates a wide range of common bugs arising in both streaming and parallel sequence processing, and can be implemented in any deep learning library. A composable and declarative API, along with a comprehensive suite of layers and combinators, streamlines the construction of production-scale models from simple streamable components while preserving strong correctness guarantees. Our current implementations of SequenceLayers (JAX, TensorFlow 2) are available at https://github.com/google/sequence-layers.
BibTeX
@misc{skerryryan2025sequencelayers,
  author = {R. J. Skerry-Ryan and Julian Salazar and Soroosh Mariooryad and David Kao and Daisy Stanton and Eric Battenberg and Matt Shannon and Ron J. Weiss and Robin Scheibler and Jonas Rothfuss and Tom Bagby},
  title = {{SequenceLayers: Sequence Processing and Streaming Neural Networks Made Easy}},
  year = {2025},
  eprint = {2507.23292},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.23292},
}
Figure