Latent State Propagation Across Sequence Boundaries

52 concepts43 papers1 proposal

When processing long sequences—whether a hours-long audio stream, a multi-turn conversation, or a book-length document—machine learning models inevitably run into memory walls. Standard Recurrent Neural Networks (RNNs), Transformers, and State Space Models (SSMs) cannot process infinitely long inputs in a single forward pass due to hardware constraints.

The conventional workaround is to slice the long sequence into smaller, manageable chunks or mini-batches. However, if we reset the model's internal hidden state to zero at the beginning of each new chunk, we introduce context fragmentation. The model becomes blind to anything that occurred before the current boundary.

Latent state propagation solves this by carrying over the final hidden state of one segment to serve as the initial state for the next. Instead of treating each batch or segment as an independent entity, the model maintains a continuous trajectory through its latent space.

Standard Training (Zero-Initialization):
[Segment 1] -> h_last1 ✕ (Reset to 0) -> [Segment 2] -> h_last2 ✕ (Reset to 0) -> [Segment 3]

Latent State Propagation:
[Segment 1] -> h_last1 ───────────────> [Segment 2] -> h_last2 ───────────────> [Segment 3]

By persisting hidden states across boundaries, models can capture long-range dependencies that far exceed their immediate training window, all while maintaining a constant memory footprint.


The Unexplored States Hypothesis

To understand why latent state propagation works, we must first look at why models fail without it. Modern recurrent architectures like Mamba or Linear Attention theoretically possess an infinite context window because their recurrent updates compress history into a fixed-size state. Yet, in practice, they often fail to generalize to sequences longer than those seen during training.

Buitrago et al. explain this failure mode through the Unexplored States Hypothesis. When a model is trained exclusively with zero-initialized hidden states on short segments, it only learns how to process information from a "cold start." As a sequence grows longer during inference, the hidden state distribution shifts away from this initial zone, entering "mature" regions of the state space that the model never encountered during training.

By implementing inter-sequence state initialization—passing the final state of a previous sequence into the next—the authors force the model to learn how to process inputs regardless of its starting point in the state space. This simple intervention allows models to generalize to sequence lengths orders of magnitude longer than their training context.

Similarly, in long-form speech recognition, Narayanan et al. introduced Random State Passing (RSP). By carrying over internal neural network states between consecutive mini-batches during training, the model learns to remain stable and accurate when encountering non-zero internal states, drastically reducing word error rates on extended recordings.


Segment-Level Recurrence and Caching

In the Transformer domain, processing long sequences requires breaking the quadratic complexity of self-attention. Rather than computing attention across an entire document, models can apply attention to localized blocks while passing historical context forward via segment-level recurrence.

In XLNet, Yang et al. utilize a state-reuse mechanism where the hidden states computed for a previous segment are cached and reused as an extended context for the current segment. This allows the network to look back beyond its fixed attention window without recomputing the representations of past tokens.

Segment t-1:  [ x_1, x_2, x_3 ] ───> Cache Hidden States (h_prev)
                                           │
                                           ▼
Segment t:    [ x_4, x_5, x_6 ] ───> Attend to [ h_prev || current_inputs ]

This concept scales across modalities. For video paragraph captioning, Lei et al. developed the Memory-Augmented Recurrent Transformer (MART), which updates a persistent memory state at the boundary of each video segment or sentence. This segment-level recurrence tracks the overarching narrative arc, preventing the repetitive or incoherent outputs typical of isolated chunk processing. For long document summarization under strict hardware constraints, Cao et al. demonstrate that partitioning text into manageable chunks while preserving context through external memory buffers allows consumer-grade GPUs to process documents spanning hundreds of thousands of tokens.


Eliminating Truncation Bias in Optimization

The benefits of state propagation extend to how gradients are calculated in unrolled computation graphs. When training recurrent models on long sequences, researchers typically rely on Truncated Backpropagation Through Time (TBPTT), which cuts the computation graph into short segments to save memory.

However, resetting states or ignoring long-term dependencies at segment boundaries introduces severe truncation bias. Schiller et al. demonstrate that standard zero-initialization of hidden states for each data chunk creates "initialization transients" that degrade performance, proving that propagating hidden states across sequence boundaries provides stronger mathematical guarantees for stateful training regimes.

To compute completely unbiased gradients in these truncated graphs, Vicol et al. proposed Persistent Evolution Strategies (PES). Instead of resetting random perturbations at the end of a short segment, PES propagates these perturbations continuously across sequential segments. This persistence ensures that the long-term influence of parameters on future states is captured accurately, allowing the model to navigate chaotic loss landscapes without the memory overhead of full unrolling.


Streaming Inference and Real-Time Processing

For real-time applications like live speech transcription or keyword spotting, a model cannot wait for an entire sequence to finish before generating an output. It must process data in a streaming fashion, chunk by chunk.

In the Neural Transducer framework proposed by Jaitly et al., inputs are processed in discrete temporal windows. The authors show that maintaining the hidden state across these blocks is critical; removing inter-block recurrence causes error rates to spike, as local windows lack the historical context required to disambiguate current inputs.

This stateful continuity is vital when deploying models to production:

  • Keyword Spotting: Rybakov et al. emphasize that stateful training is essential for maintaining model accuracy when transitioning from fixed-length training clips to continuous, live audio streams on mobile devices.
  • Online Speech Recognition: Hwang et al. introduce a partial windowing approach that allows unidirectional RNNs to be trained on infinitely long data streams by persisting recurrent states across concatenated sequence boundaries. Similarly, the work on Transformer ASR with Contextual Block Processing passes a "context inheritance" vector between corresponding layers of consecutive audio blocks, allowing deep layers to integrate historical context without global attention overhead.

Tracking Latent Flow in Dialogue

In conversational AI, a common approach to maintaining context is to concatenate raw text from previous dialogue turns and feed it back into the model. However, this approach quickly floods the model's context window with redundant tokens.

Latent state propagation offers a cleaner alternative: representing the conversation as a continuous flow of latent semantics. Instead of re-evaluating the raw text history at every turn, the model passes its internal hidden representations forward.

Raw Text Concatenation (Inneficient):
Turn 1: [Q1 + A1]
Turn 2: [Q1 + A1 + Q2 + A2]
Turn 3: [Q1 + A1 + Q2 + A2 + Q3 + A3]  <-- Context window grows quadratically

Latent State Propagation (Efficient):
Turn 1: [Q1] ───> h_latent1 ───┐
                               ▼
Turn 2:                  [Q2 + h_latent1] ───> h_latent2 ───┐
                                                            ▼
Turn 3:                                               [Q3 + h_latent2]

Huang et al. demonstrated this with FlowQA, a mechanism that propagates latent reasoning states across a sequence of questions. This allows the model to retain the internal reasoning and semantics developed during prior turns to better inform the current context. Chen et al. extended this to graph structures with GraphFlow, passing latent reasoning states through dynamically evolving text graphs.

As surveyed by Zaib et al., treating a multi-turn dialogue as a continuous flow of latent states mirrors how humans track conversational context—preserving deep semantic insights without the computational burden of re-processing the past.

Key Foundational Work

Temporal-Difference Networks with History (2005) — contributed 1 concept elements to this direction.
Credit Assignment through Time: Alternatives to Backpropagation (1993) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_latent_state_propagation_0b7d484e,
  title = {latent state propagation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-state-propagation/rdr_0b7d484e2c634913b52a211cb245d435},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_latent_state_propagation_0b7d484e_v1,
  title = {latent state propagation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-state-propagation/rdr_0b7d484e2c634913b52a211cb245d435/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake