When working with sequential models like Recurrent Neural Networks (RNNs), LSTMs, or state-space layers, a foundational question arises before the first token is even processed: What is the initial state of the network's memory?
Historically, the default choice has been zero-initialization ($h_0 = \vec{0}$). However, as Fabius et al. point out, an arbitrary zero-state completely fails to incorporate the underlying data distribution or specific characteristics of the input. It forces the network to begin in a state of total amnesia, leading to slower convergence and suboptimal performance because the model must waste its first few steps "warming up" to establish context.
To build better intuition, think of the hidden state not just as a passive scratchpad, but as a task-specific prior. By explicitly computing $h_0$ from static metadata, preceding sequences, or a separate initializer network, we can prime the model to interpret sequential transitions through the lens of that context from the very first step.
Conditioning on Static Metadata and Priors
When we want a sequence model to account for global or static context, a naive approach might concatenate this metadata to every single input token. However, Rastogi et al. demonstrated that initializing hidden states with a context vector is significantly more effective and computationally efficient than token-level concatenation. It shapes the interpretation of the entire sequence from the outset rather than acting as a repetitive feature.
This early fusion mechanism allows the model to immediately anchor its representations. For instance, Zhuang et al. use compressed attribute representations to initialize LSTMs for attributed sequences, outperforming late-stage concatenation.
In healthcare, Marx integrates static patient characteristics (like age and BMI) into the initial hidden state for blood glucose forecasting, while Li et al. use a secondary network to encode cohort-level features into the initial states of an encoder-decoder framework to generalize across heterogeneous patient populations. To improve prediction accuracy in state-dependent scenarios, Fioretto et al. incorporate a previously known valid state as a contextual input, leveraging structural similarities to refine current predictions. Even in general time-series forecasting, Kraus et al. utilize a learnable embedding token to initialize internal states, allowing the model to smoothly transition from a generic prior to a dataset-specific distribution.
Sequential and Hierarchical State Passing
When dealing with multi-step reasoning or long documents, context-aware initialization provides a natural bridge between distinct sequences. Instead of resetting the hidden state between related inputs, the final hidden state of a primary encoder becomes the initial state of a secondary encoder.
This conditional encoding ensures that the second sequence is interpreted entirely within the context established by the first. For example, in Recognizing Textual Entailment, Rocktäschel et al. process a premise first and use its final internal state to initialize the hypothesis encoder. Similarly, in stance detection, Augenstein et al. and the authors of Cross-Target Stance Classification with Self-Attention Networks initialize text encoders using the final states of a target-specific encoder, ensuring target-dependent feature extraction from the start.
[Primary Encoder] ---> (Final Hidden State) ---> [Secondary Encoder]
(Processes Target/Premise) (Initial State h_0)
|
v
(Processes Text/Hypothesis)
This principle extends naturally to broader discourse and hierarchical modeling:
- Cross-Sentence Dependencies: Luo and Glass transfer context between adjacent sentences for co-reference resolution without the computational burden of processing an entire document as a single continuous sequence.
- Discourse Context: Ghosh et al. and Bernardy et al. leverage preceding conversation or document context to initialize states for sarcasm analysis and sentence acceptability judgments.
- Thematic Consistency: Chen and Qi initialize sentence encoders with document-wide interactions, and Chen et al. use a concise summary (like a paper title) to initialize global memory states, ensuring the model retains the primary topic.
- Hierarchical Optimization: Li et al. optimize speech separation by using global context layer outputs to initialize local processing units, conditioning local feature extraction on the broader temporal context.
Initializer Networks and Warm Starts
In physical and dynamic systems, waiting for a recurrent model to accumulate context through a "washout" or warmup period is often impractical or costly. Instead, researchers deploy specialized initializer networks that map the initial configuration or recent history directly to the hidden and cell states.
In autonomous driving, Gibson et al. replace traditional washout periods with an initializer network that maps recent historical data directly to LSTM states, enabling immediate, accurate trajectory generation without wasting initial data points. For real-time human motion tracking and character locomotion, Yi et al. and Harvey et al. predict initial hidden states from the starting configuration or frame, preventing the system from losing track during static periods or generating disjointed transitions.
This strategy is also vital in computer vision and inverse problems. Yang et al. compute a recurrent tracker's initial state from the first frame's appearance to lock onto a target immediately, while Yiasemis et al. process low-quality inputs to initialize a recurrent network for accelerated MRI reconstruction, ensuring the iterative optimization begins with a context-aware representation.
Even in reinforcement learning, when agents are teleported or initialized at intermediate states, Wan et al. populate their recurrent hidden states using the history of synthetically generated trajectories. This continuity provides the necessary temporal context required for consistent decision-making from the new starting point.
Extension to Attention-Based Architectures
While state initialization is natively a recurrent concept, it directly maps to modern attention-based architectures that process data in blocks or act as recurrent systems.
In streaming speech recognition, Tsunoo et al. show that the performance of state-passing mechanisms is highly sensitive to initialization; combining positional encodings with a summary of the current input block provides a superior initial state for tracking sequence positions.
Remarkably, this perspective bridges the gap between recurrent models and Transformers. Chen et al. demonstrate that linearized-attention Transformers can be mathematically interpreted as recurrent systems where In-Context Learning (ICL) prompts are stored precisely as an initial hidden state. By modifying internal states or bias terms to match this initialization, context becomes a permanent, zero-cost part of the model's architecture, effectively treating in-context learning as a weight initialization problem.
Key Foundational Work
An Empirical Evaluation of various Deep Learning Architectures for Bi-Sequence Classification Tasks. (2016) — contributed 2 concept elements to this direction.
Reasoning about Entailment with Neural Attention Tim Rocktäschel (2016) — contributed 2 concept elements to this direction.
Stance Detection with Bidirectional Conditional Encoding (2016) — contributed 1 concept elements to this direction.