Autoregressive Large Language Models (LLMs) owe their generative prowess to the next-token prediction objective. To ensure that a model predicts a token using only preceding context, transformers employ a causal attention mask—a lower-triangular matrix that zeroes out attention scores for all future tokens. While this constraint is essential for generation, it introduces a fundamental bottleneck when the same architecture is tasked with understanding or representing text globally.
When a model processes a sequence causally, early tokens cannot "see" later tokens. This creates a severe positional bias where information at the end of a sequence cannot influence the representations of tokens at the beginning. In a theoretical analysis, Chris Hays et al. proved in The Impossibility of Inverse Permutation Learning in Transformer Models that standard decoder-only transformers are fundamentally incapable of learning global data transformations, such as inverse permutations, because information cannot flow backward through the causal mask. As a result, adapting these massive, pre-trained generative models for discriminative or representational tasks requires a structural shift: removing the causal mask to enable bidirectional attention.
The Mechanics of Mask Conversion
The transition from a causal decoder to a bidirectional encoder is conceptually straightforward but profoundly alters the model's internal dynamics. By removing the upper-triangular restriction from the attention matrix during fine-tuning or inference, every token in a sequence is permitted to attend to every other token, regardless of its position.
This modification completely changes how word representations are formed. In a semantic probing study, Zhaoxin Feng et al. demonstrated in Learning to Look at the Other Side: A Semantic Probing Study of Word Embeddings in LLMs with Enabled Bidirectional Attention that enabling bidirectional attention grants the model "foresight"—the ability to resolve lexical ambiguities using future context—at the cost of some generative "hindsight." To successfully navigate this trade-off, models are typically fine-tuned using contrastive learning or masked token reconstruction, allowing them to synthesize holistic semantic representations rather than strictly sequential ones, as detailed in a comprehensive survey on the topic (When Text Embedding Meets Large Language Model: A Comprehensive Survey).
Turning Generative Models into Generalist Encoders
Rather than training bidirectional models (like BERT) from scratch, a rapidly growing body of work focuses on converting pre-trained decoder-only models into high-performance embedding and retrieval engines.
A prime example of this is the GritLM framework introduced by Niklas Muennighoff et al. in Generative Representational Instruction Tuning. By using specialized instructions, the model dynamically toggles its attention mask: it retains the causal mask for text generation but drops it entirely during the embedding phase, allowing the model to function as a full bidirectional encoder. Similarly, Chankyu Lee developed NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models, showing that removing the causal mask during contrastive training allows generative models to capture full semantic context without requiring specialized pre-training objectives.
This bidirectional conversion strategy has proven highly effective across several specialized retrieval and representation paradigms:
- Dense and Sparse Retrieval: Hansi Zeng et al. demonstrated in Scaling Sparse and Dense Retrieval in Decoder-Only LLMs that replacing the causal mask allows decoder-only architectures to excel at both dense and sparse retrieval tasks. For lexicon-based embeddings, Tao Shen et al. utilized bidirectional attention in Enhancing Lexicon-Based Text Embeddings with Large Language Models to accurately weight lexical features using global context.
- Sequential Recommendation: In LLM2Rec: Large Language Models Are Powerful Embedding Models for Sequential Recommendation, Xiaohao Liu et al. stripped the causal mask to capture the comprehensive context of item descriptions, outperforming traditional unidirectional processing.
- Model Distillation and Efficiency: The authors of DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers found that enabling bidirectional attention during the training of pruned decoder models significantly enhances their retrieval performance. Furthermore, Xinping Zhao et al. showed in KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model that even compact 0.5B parameter models can rival much larger architectures when converted to bidirectional sentence encoding.
Mitigating the Reversal Curse
The limitations of causal masking are not confined to embedding quality; they also manifest as logical failures in reasoning. The most famous of these is the "Reversal Curse," where an LLM trained on the fact "Entity A is related to Entity B" fails to realize that "Entity B is related to Entity A." Because the next-token prediction objective only optimizes information flow in one direction, the reverse association is never structurally encoded.
Bidirectional attention provides a direct architectural cure for this limitation. Ang Lv et al. introduced BICO in An Analysis and Mitigation of the Reversal Curse, a fine-tuning framework that removes the causal triangular mask to grant models bidirectional context, allowing them to capture symmetric relationships between entities.
This link between bidirectional information flow and logical symmetry is further supported by alternative model families. Sangwoo Shin et al. investigated this in Understanding the Reversal Curse Mitigation in Masked Diffusion Models through Attention and Training Dynamics, revealing that Masked Diffusion Models (MDMs) are naturally resilient to the Reversal Curse. Because MDMs rely on a transformer encoder architecture with bidirectional attention, they create symmetric internal representations that automatically couple relational facts regardless of their sequence order.
Non-Causal Processing in Multimodal and Generative Tasks
While text has a natural sequential order, visual data does not. Forcing vision-language models (VLMs) to process visual tokens under strict causal constraints introduces artificial limitations. Consequently, researchers are increasingly relaxing or removing causal masks in multimodal settings.
In Rethinking Causal Mask Attention for Vision-Language Inference, the authors demonstrated that allowing visual tokens to "preview" future context during the prefill stage significantly improves reasoning on complex tasks without requiring any additional training. For visual document retrieval, Gabriel de Souza P. Moreira et al. showed in Nemotron ColEmbed V2: Top-Performing Late Interaction embedding models for Visual Document Retrieval that converting generative decoder VLMs into retrieval encoders via bidirectional attention drastically improves document matching accuracy.
Even within generative tasks, bidirectional attention is being used to overcome the positional biases of causal modeling:
- Prompt Encoding: Bingqi Ma et al. discovered in Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models that causal masking causes text-to-image generators to ignore details at the end of long prompts. Replacing this with bidirectional attention ensures that information at the end of a sequence is just as influential as the beginning.
- Iterative Refinement: In FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities, Jin Wang et al. replaced autoregressive generation with discrete flow matching. Their non-causal architecture uses bidirectional attention during denoising, allowing tokens to interact globally to facilitate layout planning and attribute binding. Similarly, Tianhong Li et al. utilized bidirectional attention in the encoding stage of continuous-valued token sequence modeling in Autoregressive Image Generation without Vector Quantization to improve latent representations.
- Unified Architectures: A. J. Piergiovanni et al. showed in MaMMUT: A Simple Architecture for Joint Learning for MultiModal Tasks that a text decoder utilizing bidirectional masking significantly outperforms standard causal masking on cross-modal retrieval tasks by forming superior global text-image representations.
Architectural Exploration and Future Outlook
As the field moves toward unified foundation models capable of both generation and representation, understanding bidirectional information flow becomes a theoretical necessity. Jiechao Yang et al. pointed out in ETAS: Zero-Shot Transformer Architecture Search via Network Trainability and Expressivity that traditional zero-shot proxy metrics often fail to account for non-causal attention mechanisms due to a lack of theoretical derivations for bidirectional information flow. Extending these mathematical frameworks is a crucial next step for automated architecture search.
Ultimately, the shift toward bidirectional attention conversion demonstrates that the boundary between "encoders" (like BERT) and "decoders" (like GPT) is largely artificial. By dynamically manipulating the attention mask, a single foundation model can seamlessly transition between a sequential generator and a global encoder, maximizing both its expressivity and its utility.
Key Foundational Work
LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (2024) — contributed 2 concept elements to this direction.
Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models (2024) — contributed 2 concept elements to this direction.
When Text Embedding Meets Large Language Model: A Comprehensive Survey (2024) — contributed 1 concept elements to this direction.