How do we teach a machine to look at an image and describe what it sees? For years, computer vision treated images as static grids of pixels to be classified, while natural language processing treated sentences as discrete sequences of tokens. The breakthrough that bridged these two worlds was the realization that generating a description from an image can be framed as a translation problem. Instead of translating French to English, we can translate pixels to tokens.
This vision-to-language pipeline relies on the multimodal encoder-decoder architecture. By pairing a visual encoder with a sequential language decoder, these models map high-dimensional spatial data into a shared semantic space, enabling machines to generate fluent, contextually accurate natural language from visual inputs.
The Core Mechanism: Translating Pixels to Vectors to Tokens
The foundational architecture of modern vision-to-language systems relies on a two-part pipeline: a visual encoder and a linguistic decoder. As surveyed by Desmond Elliott et al., the transition from rigid, rule-based templates to neural encoder-decoder networks fundamentally changed the field by allowing models to learn an end-to-end mapping from raw pixels to natural language.
[ Visual Input ] ---> ( Encoder: CNN / ViT ) ---> [ Latent Vector / Grid ]
|
v
[ Generated Text ] <-- ( Decoder: RNN / Transformer ) <----+
1. The Encoder (Compressing Spatial Information)
An image contains millions of pixels, but relatively low semantic density. The encoder's job is to compress this high-dimensional spatial data into a dense, low-dimensional feature vector that captures the core entities and relationships within the scene.
In early canonical models, such as the Neural Image Caption (NIC) framework introduced by Oriol Vinyals et al., a deep Convolutional Neural Network (CNN) pre-trained on image classification served as the encoder. The final classification layer is discarded, and the output of the last pooling or fully connected layer is extracted as a fixed-length visual summary. As described by Andrew Gilbert et al. and Yongqing Zhu et al., this vector acts as a complete semantic blueprint of the visual input.
2. The Decoder (Unrolling the Sequence)
The language decoder takes the encoder's visual blueprint and translates it into a sequence of words. This process is autoregressive: the model predicts one token at a time, conditioning its next prediction on both the visual context and all previously generated tokens.
In traditional setups, the visual vector is used to initialize the hidden state of a Recurrent Neural Network (RNN) or a Long Short-Term Memory (LSTM) network. Ferdous Sohel et al. explain that this treats image captioning as a sequence-to-sequence problem. At step $t$, the decoder ingests the token from step $t-1$ and updates its internal hidden state, maintaining a running memory of the sentence structure while remaining anchored to the initial visual context.
Overcoming the Information Bottleneck: Attention
Squeezing an entire image—with all its objects, attributes, and spatial relationships—into a single, fixed-length vector creates an information bottleneck. Forcing a model to compress a complex scene into a single vector means fine-grained details are inevitably lost.
To solve this, researchers introduced spatial attention mechanisms. Instead of collapsing the encoder's final layers into a single vector, the decoder is given access to a grid of localized feature vectors representing different regions of the image.
When generating a specific word (e.g., "dog"), the decoder computes an attention score across the spatial grid, effectively "looking" at the region containing the dog. As demonstrated by Ting-Yao Hsu et al. in the context of scientific figure captioning, this allows the decoder to dynamically weight different spatial regions at each step of the sequence generation.
Decoder Step 1: "A" ---> Focuses on global context
Decoder Step 2: "brown" ---> Focuses on spatial region [Row 2, Col 3] (Texture)
Decoder Step 3: "dog" ---> Focuses on spatial region [Row 2, Col 3] (Object)
Decoder Step 4: "jumping" ---> Focuses on spatial region [Row 1, Col 2] (Action)
The Modern Shift to Transformers
The encoder-decoder paradigm has evolved beyond CNN-RNN pairings by adopting Transformer architectures for both vision and language.
- Vision Transformers (ViTs) replace CNNs by slicing an image into a sequence of patches, treating them much like words in a sentence.
- Transformer Decoders replace RNNs, utilizing self-attention to process text sequences in parallel during training and cross-attention to map directly to the visual patch embeddings.
For instance, Benjamin Hou et al. developed a hybrid architecture that flattens spatial image features into a sequence, allowing a Transformer decoder to apply attention across image regions as if they were linguistic tokens. Similarly, Qing Jiang et al. utilize a ViT backbone paired with a Transformer decoder to capture global dependencies across images for scene text recognition.
To balance representation quality with computational overhead, researchers like Alexandru-Cosmin Mihai have shown that combining a contrastively pre-trained vision transformer (such as CLIP) with a lightweight LSTM decoder can yield highly efficient temporal modeling and text generation without the heavy training footprint of full autoregressive language models.
Adapting to Temporal Data: Video Captioning
The encoder-decoder framework adapts naturally from static images to temporal sequences like video. The primary challenge in video captioning is that the encoder must capture not only spatial features (what objects are in the frame) but also temporal features (how those objects move over time).
As detailed by Zuxuan Wu et al., video captioning architectures typically use a 2D or 3D CNN to extract features from individual frames or short video clips. These frame-level features are then aggregated—often using an intermediate recurrent network or a temporal pooling layer—to create a unified video representation. The decoder, as shown by Min Gao 0001 et al., then unrolls this multi-layered visual representation into a coherent natural language narrative that accounts for actions taking place across time.
Specialized Domains and Structured Outputs
The utility of the vision-to-language encoder-decoder extends far beyond standard image captioning. Because the architecture treats sequence generation as an arbitrary translation task, the target sequence does not have to be standard natural language sentences; it can be structured code, medical reports, or symbolic notation.
Medical Report Generation
In clinical settings, models must translate complex medical imagery into structured diagnostic narratives. Sonit Singh demonstrated how multi-modal encoder-decoder frameworks can automate the generation of radiology reports from complex medical scans.
To ensure clinical accuracy, models like the one proposed by Chao-Han Huck Yang incorporate keyword-driven generation to ground the recurrent decoder in both global visual features and expert-defined clinical terms. This ensures that the sequential output remains strictly tied to verified visual anomalies.
Document Structure and Markup Generation
Instead of outputting prose, the decoder can be trained to output structural markup languages. The TableBank project demonstrates this by using an encoder-decoder architecture with attention to map images of document tables directly into HTML or LaTeX markup strings. The model learns to translate visual lines and cell boundaries into hierarchical row and column tags, treating layout recognition purely as a visual-to-symbolic translation problem.
Optical Music Recognition
A similar principle applies to reading sheet music. Eelco van der Wel et al. developed a convolutional sequence-to-sequence model that processes overlapping patches of sheet music images and decodes them directly into digital music notation symbols. This bypasses the need for explicit symbol segmentation by allowing the recurrent decoder to learn the implicit alignment between spatial note positions and sequential musical syntax.
Key Foundational Work
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures (2016) — contributed 2 concept elements to this direction.
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (2015) — contributed 1 concept elements to this direction.
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures (2016) — contributed 1 concept elements to this direction.