Bidirectional Attention for Decoder Only Representation

This direction is split into other directions: bidirectional attention conversion. Its last version stays readable and citable here.

Version 1 of 1121 papers151 conceptsfrozen 2026-10-04

The dominant paradigm in modern NLP is the decoder-only Transformer. Models like Llama and GPT are trained with a "causal mask"—a triangular constraint that ensures each token can only attend to the tokens that came before it. While this is essential for autoregressive generation (predicting the next word), it is fundamentally suboptimal for representation tasks like text retrieval, classification, or embedding.

Recent research has converged on a simple but powerful insight: the causal mask is not an inherent property of the model's weights, but a toggle. By removing this mask during fine-tuning or inference, we can transform generative "chatbots" into state-of-the-art bidirectional encoders.

The Limitation of Hindsight

In a standard causal model, the representation of the first word in a sentence is "blind" to the rest of the text. As Chankyu Lee and colleagues note in NV-Embed, this creates an inherent recency bias where the final tokens carry the most contextual weight, while early tokens remain impoverished.

For generation, this is fine; you can't look at words you haven't written yet. But for understanding, it is a "reversal curse." Kaiyi Zhang demonstrated in Are We Falling in a Middle-Intelligence Trap? that causal models often fail to realize that if "A is B," then "B is A," because the information flow is strictly one-way. By enabling bidirectional attention, tokens can "look ahead," allowing the model to resolve lexical ambiguities and capture global semantic relationships that are invisible to a unidirectional scan.

Unmasking the Decoder

The transition from a generative model to a representational one involves replacing the causal mask with an "all-ones" mask. This allows every token in a sequence to attend to every other token simultaneously.

This architectural shift has been applied across several domains:

  • Text Embeddings: Hieu Man and the team behind ULLME showed that simply applying a bidirectional mask to a decoder-only model significantly enhances its performance in dense passage retrieval, bridging the gap between generative prowess and discriminative accuracy.
  • Code Search: In CodeXEmbed, Ye Liu demonstrates that bidirectional context is critical for capturing the full semantic meaning of code blocks, where a variable's meaning often depends on its usage later in the script.
  • Token-Level Tasks: Zongxi Li showed in LS-LLaMA that "unmasking" allows a 7B parameter model to outperform established bidirectional encoders like BERT and RoBERTa on tasks like Named Entity Recognition (NER).

The Best of Both Worlds

A common concern is whether enabling bidirectional attention "breaks" the pre-trained knowledge of a causal model. However, evidence suggests that the knowledge is already there; the mask simply prevents the model from accessing it holistically.

Niklas Muennighoff introduced a unified approach called Generative Representational Instruction Tuning (GRIT). By using specific instructions to toggle between causal (for generation) and bidirectional (for embedding) modes, a single model can excel at both. This suggests that the distinction between "encoder" and "decoder" is increasingly a matter of software configuration rather than structural necessity.

Beyond Text: Vision and Generation Order

The utility of bidirectional "unmasking" extends beyond standard NLP. In Frozen Transformers in Language Models Are Effective Visual Encoder Layers, Ziqi Pang found that removing causal masks allows LLM blocks to be repurposed as high-quality visual encoders for 3D point clouds and video.

Furthermore, this flexibility allows for non-linear generation. The Insertion Transformer by Mitchell Stern and the work of Junlong Gao on Masked Non-Autoregressive Image Captioning use bidirectional attention to generate sequences in parallel or via insertions. By seeing the "global context" of the partially constructed sequence, these models can make more informed decisions about where to place the next word, rather than being forced into a strict left-to-right straightjacket.

Conclusion

The move toward bidirectional decoder-only models represents a maturation of the Transformer architecture. By treating the attention mask as a flexible tool rather than a rigid constraint, researchers are unlocking the latent representational power of Large Language Models, allowing them to serve as both the "writer" and the "librarian" of the AI era.

Papers in this version

Showing 60 of 121 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bidirectional_attention_for_decoder_only_ec43a623,
  title = {Bidirectional Attention for Decoder Only Representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-attention-for-decoder-only-representation/rdr_ec43a6238cf1476793f8890d50dbdfe0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_bidirectional_attention_for_decoder_only_ec43a623_v1,
  title = {Bidirectional Attention for Decoder Only Representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-attention-for-decoder-only-representation/rdr_ec43a6238cf1476793f8890d50dbdfe0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}