Single Stream Sequence Fusion for Multimodal Transformers

817 concepts568 papers8 proposals

In traditional multimodal deep learning, different types of data—like text, images, and audio—are treated as fundamentally distinct entities. A typical architecture uses a specialized vision network to process images, a language model to process text, and a late-stage fusion mechanism (such as cross-attention layers or pooling) to force the two streams to interact at the very end of the pipeline.

Single-stream (or early-fusion) architectures completely flip this paradigm. Instead of maintaining separate pipelines, they ask a radical question: What if we treat every modality exactly like text?

By flattening images, audio, and actions into a sequence of standardized tokens and concatenating them into a single stream, a single Transformer backbone can process everything simultaneously. This approach eliminates the need for task-specific heads or complex late-fusion modules, allowing the network to learn deep, cross-modal interactions from the very first layer.


The Intuition of a Shared Token Space

To understand why this works, we must look at how Transformers process data. A Transformer doesn't inherently know what a "word" or a "pixel" is; it merely operates on a sequence of continuous vectors (embeddings) in a high-dimensional space.

If we can project non-textual data into this exact same embedding dimension, we can interleave modalities within a single sequence.

Dual-Stream (Late Fusion):
[Text]  ---> [Text Encoder]  ---\
                                 ===> [Cross-Attention / Fusion]
[Image] ---> [Vision Encoder] ---/

Single-Stream (Early Fusion):
[Text Tokens + Image Patches] ---> [Shared Transformer Backbone] ---> Multimodal Output

Early implementations of this concept focused on combining text with localized visual regions. For example, Alberti et al. introduced B2T2, a model that embeds visual bounding box features directly into the initial input sequence of a BERT architecture. By treating visual objects as tokens equivalent to words, the self-attention mechanism could immediately map linguistic entities to physical objects.

Similarly, Li et al. analyzed VisualBERT, demonstrating that when text tokens and image regions are processed simultaneously within a single transformer stack, individual attention heads naturally learn to align words with their corresponding image regions without any explicit grounding supervision.


Why Early Fusion Works: Deep Self-Attention

The core advantage of a single-stream architecture lies in the mechanics of self-attention. In a dual-stream model, visual features are computed independently of textual context. If an image contains a ambiguous object, the vision encoder must guess its identity without the help of the accompanying text prompt.

In a single-stream model, every single layer's self-attention mechanism can look at both text and visual tokens. This allows for what Jang et al. describe as cross-modal steering. By concatenating image patches and text tokens into a shared encoder, linguistic cues can directly shift the model's internal attention toward specific visual features progressively throughout the network. The text helps the model "see" the image, and the image helps the model "understand" the text at every step of computation.

This deep integration is equally powerful for generative tasks. Instead of using separate cross-attention layers to condition an image generation model on text, Tang et al. demonstrated that concatenating text and image tokens into a shared self-attention block at every layer allows continuous, bidirectional interaction between the language model's internal representations and the image synthesis process.


Generalizing to Any Modality

Once you accept that any data type can be flattened into a sequence of vectors, the single-stream architecture becomes entirely modality-agnostic.

  • Audio-Visual Fusion: Kim et al. developed AVCap, which treats audio and video features as text tokens within a Large Language Model (LLM). A specialized causal attention mask ensures that while text generation remains autoregressive, every generated word can attend to the full context of the preceding audio-visual tokens.
  • Robotics and Action Histograms: In robotic manipulation, the authors of Instruction-driven history-aware policies for robotic manipulations represented language instructions, multi-view visual patches, and historical action sequences as a single unified sequence of tokens, allowing standard attention to learn complex dependencies across time and sensory streams.
  • Pixels Only: Taking this concept to its logical extreme, Tschannen introduced CLIPPO, showing that even text can be rendered as pixels and fused directly into a single visual input stream. This proved that explicit architectural components for text tokenization are not strictly necessary for cross-modal reasoning.

The Trade-offs of a Single Stream

Despite its conceptual elegance and deep fusion capabilities, the single-stream approach introduces significant engineering challenges:

  1. Sequence Length Explosion: The primary bottleneck of standard Transformers is that self-attention complexity scales quadratically ($O(N^2)$) with sequence length. Flattening a high-resolution image into $16 \times 16$ patches yields hundreds of tokens, which quickly overwhelms the context window when concatenated with long text sequences.
  2. Attention Collapse: Forcing a model optimized for discrete text tokens to process continuous visual or auditory signals can destabilize training. Wang et al. highlighted an "attention collapse" issue when adapting causal text decoders (like LLaMA) to process visual sequences, requiring specialized masking and token strategies to stabilize training.
  3. Superficial Grounding: Sun et al. noted that simply treating visual features as tokens might allow a model to associate patches with words via statistical correlation without necessarily forcing a fundamental restructuring of how physical concepts are represented internally.

By shifting the complexity away from custom architectural wiring and placing it entirely onto the data representation layer, single-stream transformers leverage the raw scaling power of the Transformer backbone to solve multimodal alignment.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_unified_multimodal_transformer_467d1f91,
  title = {unified multimodal transformer},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unified-multimodal-transformer/rdr_467d1f91cad84c37afc398ff92d14394},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_unified_multimodal_transformer_467d1f91_v1,
  title = {unified multimodal transformer},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unified-multimodal-transformer/rdr_467d1f91cad84c37afc398ff92d14394/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake