When a sequence-to-sequence model generates text or actions based on an image, the standard approach often initializes the recurrent neural network (RNN) hidden states to zero. This forces the model to begin generation with a blank slate, relying entirely on subsequent attention mechanisms to pull relevant visual information. Fan Lyu et al. describe this in Coarse to Fine: Multi-label Image Classification with Global/Local Attention as a "blind search" problem: the model lacks an initial understanding of the scene's content, which can lead to poor early predictions.
To resolve this, a prominent line of research focuses on visual state initialization—using features extracted from an image to seed the internal memory of the sequence generator before it processes or generates the first token. Instead of starting from zero, the model's initial hidden state $h_0$ is calculated directly from global visual features, as demonstrated by Iacer Calixto et al. in DCU System Report on the WMT 2017 Multi-modal Machine Translation Task and Desmond Elliott et al. in Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description. This provides a holistic visual prior that grounds the network's internal representation from the very beginning of the encoding or decoding process.
Hidden States vs. Token Injection
How exactly should visual information be introduced at the start of a sequence? Researchers have converged on two primary mechanisms:
- Hidden State Conditioning ($h_0$): The visual feature vector is mapped directly into the hidden state space of the encoder or decoder. Iacer Calixto et al. found in Incorporating Global Visual Features into Attention-based Neural Machine Translation that initializing the hidden states with global visual features is more effective than treating the image as a discrete input token. This approach allows the model to integrate non-textual context without forcing the attention mechanism to explicitly select it at every step. Similarly, Pranava Madhyastha et al. showed in Defoiling Foiled Image Captions that initializing the hidden state with image-derived features allows recurrent networks to process linguistic sequences while remaining conditioned on the global or categorical context of the scene.
- Initial Token Injection ($x_0$): Instead of modifying the hidden state, the visual representation is treated as the very first token in the sequence, effectively replacing or preceding the standard start-of-sentence (
<BOS>) token. Mengye Ren et al. demonstrated this in Exploring Models and Data for Image Question Answering by treating visual features as an initial token in an LSTM, allowing the network to naturally condition a text query on the global image context. Ozan Caglayan et al. systematically compared these approaches in Multimodal Grounding for Sequence-to-Sequence Speech Recognition, showing that initializing both the encoder and decoder hidden states with a visual summary—or replacing the start token with a continuous visual vector—successfully embeds semantic context before any sequential processing begins.
Trajectory Control and Noise Mitigation
Why limit visual injection to the initial step rather than feeding it continuously? The core intuition is that continuous exposure to raw visual features can introduce redundant information or overfit to visual noise, distracting the language model from sequential dependencies.
Oriol Vinyals et al. pioneered this design choice in Show and tell: A neural image caption generator, injecting the image representation only at the initial time step so that the visual context sets the trajectory without interfering with word generation. Xuelong Li et al. expanded on this in Multi-modal gated recurrent units for image description, showing that initializing a Gated Recurrent Unit (GRU) decoder solely at the first step forces the recurrent hidden state to carry the necessary visual context throughout the entire sequence, bridging the gap between vision and language without introducing noise.
This initial "push" fundamentally alters the generation path. Haoran Li 0001 et al. observed in Multi-modal Sentence Summarization with Modality Attention and Image Filtering that incorporating visual features during initialization significantly improves the accuracy of the first generated word, which sets a more effective trajectory for the remainder of the summary. For long or complex sentences, Mingyang Zhou 0004 et al. demonstrated in A Visual Attention Grounding Neural Model for Multimodal Machine Translation that initializing a decoder with a visually-grounded text representation helps maintain thematic consistency across the entire sequence, preventing the model from losing track of the visual context over time.
Compositional and Multi-Point Initialization
As the field progressed, researchers began exploring more sophisticated initialization strategies that combine visual features with other modalities or inject them at multiple points.
- Semantic and Knowledge Augmentation: Rather than using raw visual embeddings alone, Ting Yao et al. showed in Boosting Image Captioning with Attributes that the order of initialization matters: seeding a decoder with high-level semantic attributes before introducing global image features provides a more effective starting state. Yimin Zhou et al. concatenated visual features with semantic embeddings from external knowledge graphs in Improving Image Captioning by Leveraging Knowledge Graphs, allowing the decoder to start generation with both direct visual evidence and inferred commonsense priors.
- Multi-Point Injection: To maximize grounding, Jean-Benoit Delbrouck et al. found in QIAI at MEDIQA 2021: Multimodal Radiology Report Summarization that simultaneously injecting visual features into the encoder, decoder, and hidden state initialization yields superior results compared to a single injection point, ensuring the model leverages visual context comprehensively. Mengmeng Cui et al. combined holistic image features with a global glimpse vector in Representation and Correlation Enhanced Encoder-Decoder Framework for Scene Text Recognition to initialize decoder states, providing a guided starting point.
- Modality Ordering: In multi-modal relation extraction, Qian Li et al. noted in Dual-Gated Fusion with Prefix-Tuning for Multi-Modal Relation Extraction that the sequence of modalities matters, with visual context ideally preceding textual data to prime the interpretation of subsequent text.
Beyond Text: Continuous Control and Decision Making
The intuition behind visual state initialization extends beyond language generation into sequential decision-making and continuous control.
In reinforcement learning, Amir-massoud Farahmand et al. used a computer vision model in Improving Skin Condition Classification with a Visual Symptom Checker Trained using Reinforcement Learning to generate an initial probability distribution from an image, serving as the starting state for a sequential decision agent. This visual prior narrows the search space for subsequent actions compared to starting from a uniform distribution.
Furthermore, to handle multi-frequency sensor fusion in robotics, Sumeet Singh et al. processed low-frequency visual frames in Multiscale Sensor Fusion and Continuous Control with Neural CDEs to initialize the hidden state of a continuous-time Neural Controlled Differential Equation (CDE) model. This visual seed provides a contextual starting point that conditions how the state subsequently evolves in response to higher-frequency tactile or proprioceptive inputs.
Whether translating text in low-resource settings (Koel Dutta Chowdhury et al. in Multimodal Neural Machine Translation for Low-resource Language Pairs using Synthetic Data) or controlling a robotic arm, initializing the hidden state with visual context ensures that sequential models are grounded from step zero.
Key Foundational Work
Exploring Models and Data for Image Question Answering (2015) — contributed 1 concept elements to this direction.
Bidirectional Long-Short Term Memory for Video Description (2016) — contributed 1 concept elements to this direction.
Show and tell: A neural image caption generator (2015) — contributed 1 concept elements to this direction.