Converting Causal Transformers to Bidirectional Encoders

28 concepts21 papers1 proposal

Autoregressive Large Language Models (LLMs) owe their generative prowess to the next-token prediction objective. To ensure that a model predicts a token using only preceding context, transformers employ a causal attention mask—a lower-triangular matrix that zeroes out attention scores for all future tokens. While this constraint is essential for generation, it introduces a fundamental bottleneck when the same architecture is tasked with understanding or representing text globally.

When a model processes a sequence causally, early tokens cannot "see" later tokens. This creates a severe positional bias where information at the end of a sequence cannot influence the representations of tokens at the beginning. In a theoretical analysis, Chris Hays et al. proved in The Impossibility of Inverse Permutation Learning in Transformer Models that standard decoder-only transformers are fundamentally incapable of learning global data transformations, such as inverse permutations, because information cannot flow backward through the causal mask. As a result, adapting these massive, pre-trained generative models for discriminative or representational tasks requires a structural shift: removing the causal mask to enable bidirectional attention.

The Mechanics of Mask Conversion

The transition from a causal decoder to a bidirectional encoder is conceptually straightforward but profoundly alters the model's internal dynamics. By removing the upper-triangular restriction from the attention matrix during fine-tuning or inference, every token in a sequence is permitted to attend to every other token, regardless of its position.

This modification completely changes how word representations are formed. In a semantic probing study, Zhaoxin Feng et al. demonstrated in Learning to Look at the Other Side: A Semantic Probing Study of Word Embeddings in LLMs with Enabled Bidirectional Attention that enabling bidirectional attention grants the model "foresight"—the ability to resolve lexical ambiguities using future context—at the cost of some generative "hindsight." To successfully navigate this trade-off, models are typically fine-tuned using contrastive learning or masked token reconstruction, allowing them to synthesize holistic semantic representations rather than strictly sequential ones, as detailed in a comprehensive survey on the topic (When Text Embedding Meets Large Language Model: A Comprehensive Survey).

Turning Generative Models into Generalist Encoders

Rather than training bidirectional models (like BERT) from scratch, a rapidly growing body of work focuses on converting pre-trained decoder-only models into high-performance embedding and retrieval engines.

A prime example of this is the GritLM framework introduced by Niklas Muennighoff et al. in Generative Representational Instruction Tuning. By using specialized instructions, the model dynamically toggles its attention mask: it retains the causal mask for text generation but drops it entirely during the embedding phase, allowing the model to function as a full bidirectional encoder. Similarly, Chankyu Lee developed NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models, showing that removing the causal mask during contrastive training allows generative models to capture full semantic context without requiring specialized pre-training objectives.

This bidirectional conversion strategy has proven highly effective across several specialized retrieval and representation paradigms:

Mitigating the Reversal Curse

The limitations of causal masking are not confined to embedding quality; they also manifest as logical failures in reasoning. The most famous of these is the "Reversal Curse," where an LLM trained on the fact "Entity A is related to Entity B" fails to realize that "Entity B is related to Entity A." Because the next-token prediction objective only optimizes information flow in one direction, the reverse association is never structurally encoded.

Bidirectional attention provides a direct architectural cure for this limitation. Ang Lv et al. introduced BICO in An Analysis and Mitigation of the Reversal Curse, a fine-tuning framework that removes the causal triangular mask to grant models bidirectional context, allowing them to capture symmetric relationships between entities.

This link between bidirectional information flow and logical symmetry is further supported by alternative model families. Sangwoo Shin et al. investigated this in Understanding the Reversal Curse Mitigation in Masked Diffusion Models through Attention and Training Dynamics, revealing that Masked Diffusion Models (MDMs) are naturally resilient to the Reversal Curse. Because MDMs rely on a transformer encoder architecture with bidirectional attention, they create symmetric internal representations that automatically couple relational facts regardless of their sequence order.

Non-Causal Processing in Multimodal and Generative Tasks

While text has a natural sequential order, visual data does not. Forcing vision-language models (VLMs) to process visual tokens under strict causal constraints introduces artificial limitations. Consequently, researchers are increasingly relaxing or removing causal masks in multimodal settings.

In Rethinking Causal Mask Attention for Vision-Language Inference, the authors demonstrated that allowing visual tokens to "preview" future context during the prefill stage significantly improves reasoning on complex tasks without requiring any additional training. For visual document retrieval, Gabriel de Souza P. Moreira et al. showed in Nemotron ColEmbed V2: Top-Performing Late Interaction embedding models for Visual Document Retrieval that converting generative decoder VLMs into retrieval encoders via bidirectional attention drastically improves document matching accuracy.

Even within generative tasks, bidirectional attention is being used to overcome the positional biases of causal modeling:

Architectural Exploration and Future Outlook

As the field moves toward unified foundation models capable of both generation and representation, understanding bidirectional information flow becomes a theoretical necessity. Jiechao Yang et al. pointed out in ETAS: Zero-Shot Transformer Architecture Search via Network Trainability and Expressivity that traditional zero-shot proxy metrics often fail to account for non-causal attention mechanisms due to a lack of theoretical derivations for bidirectional information flow. Extending these mathematical frameworks is a crucial next step for automated architecture search.

Ultimately, the shift toward bidirectional attention conversion demonstrates that the boundary between "encoders" (like BERT) and "decoders" (like GPT) is largely artificial. By dynamically manipulating the attention mask, a single foundation model can seamlessly transition between a sequential generator and a global encoder, maximizing both its expressivity and its utility.

Key Foundational Work

LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (2024) — contributed 2 concept elements to this direction.
Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models (2024) — contributed 2 concept elements to this direction.
When Text Embedding Meets Large Language Model: A Comprehensive Survey (2024) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bidirectional_attention_conversion_dc13c9fd,
  title = {bidirectional attention conversion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-attention-conversion/rdr_dc13c9fdebb84fcebacff0cf248647a6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_bidirectional_attention_conversion_dc13c9fd_v1,
  title = {bidirectional attention conversion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-attention-conversion/rdr_dc13c9fdebb84fcebacff0cf248647a6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake