Simulating Bidirectional Context in Causal Language Models

17 concepts12 papers1 proposal

Autoregressive, decoder-only Large Language Models (LLMs) process text from left to right. This design choice enables highly efficient causal masking during training and generation: a token at position $i$ can only attend to tokens at positions $\le i$.

However, language is inherently bidirectional. To fully understand the word "bank" in the sentence "The bank was muddy because of the river overflow," a model needs the future context provided by "muddy" and "river." Because a causal model processes "bank" before these words appear, its initial hidden representation of the token is fundamentally limited.

To overcome this "causal bottleneck" without changing the underlying architecture or undergoing expensive retraining, a powerful and surprisingly simple paradigm has emerged: simulating bidirectionality through sequence repetition and feedback loops.


The Intuition: Time Travel via Repetition

To understand why repeating an input sequence creates bidirectional context, consider the structure of the attention matrix. In standard causal attention, the matrix is strictly lower-triangular.

If we repeat an input sequence $X$ twice within a single prompt—forming the sequence $[X_1, X_2]$—the attention matrix expands into a block structure:

$$\begin{pmatrix} \text{Causal}(X_1) & 0 \\ \text{Full Attention}(X_2 \to X_1) & \text{Causal}(X_2) \end{pmatrix}$$

While $X_1$ is processed with standard causal constraints, every token in $X_2$ can look back at every token in $X_1$. This means the first token of $X_2$ can attend to the very last token of $X_1$. From the perspective of $X_2$, the future context of the sequence has already been generated and made available in the self-attention history.

Standard Causal Pass:
[The] <- [bank] <- [was] <- [muddy]   (Tokens cannot see the future)

Repeated Pass (Echo):
[The_1] <- [bank_1] <- [was_1] <- [muddy_1]
   ^          ^          ^          ^
   |----------|----------|----------|-- [The_2] (Sees the entire past pass!)

Prompt-Level Echoes

The most direct application of this mechanism is at the prompt level, requiring zero modifications to model weights or inference code.

In the context of text embeddings, standard causal models yield poor sentence representations because the final token's embedding cannot influence the representations of earlier tokens. Springer et al. et al. (Repetition Improves Language Model Embeddings) demonstrate that simply "echoing" the input text and extracting embeddings from the second occurrence significantly boosts performance. The tokens in the second pass leverage the complete context of the first pass to form a truly holistic representation. Similarly, Shang et al. et al. (Retrieval Backward Attention without Additional Training: Enhance Embeddings of Large Language Models via Repetition) utilize this exact repetition mechanism to simulate backward attention, showing that the model's internal representations natively adapt to utilize this historical context.

This trick is equally potent for complex reasoning and syntactic tasks:


Layer-Level and State Feedback Loops

While prompt-level repetition is training-free, it incurs a computational cost: doubling the sequence length quadruples the attention computation. To circumvent this, researchers have designed architectural feedback loops that pass global context across layers or execution stages rather than expanding the prompt.

Instead of repeating the raw text, Gu et al. et al. (Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs) introduce Token Prepending. They extract a summary representation of the entire sequence from a preceding transformer layer and inject it into a placeholder token at the start of the next layer. This recursive feedback loop allows subsequent layers to access global context at the beginning of the sequence with negligible computational overhead.

Layer L+1:  [Summary of Layer L] -> [The] -> [bank] -> [was] -> [muddy]
                                      ^        ^        ^        ^
                                      |--------|--------|--------|
                                      (Attends to the global summary token)

A similar philosophy underpins the work of Soni et al. et al. (Human Language Modeling), who design a Human-aware Recurrent Transformer (HaRT). The architecture extracts high-level semantic summaries from late-stage layers and feeds them back into early-stage layers of the next processing step, maintaining a long-term global context across document boundaries.

For sequence-to-sequence tasks like machine translation, this multi-stage re-encoding can be formalized across distinct execution phases:


Exploiting Pre-existing Causal Cycles

Does this simulation of bidirectionality require explicit algorithmic intervention, or do language models naturally learn to exploit repetition?

Nwadike and Inui (Library-Like Behavior In Language Models is Enhanced by Self-Referencing Causal Cycles) reveal that pre-training data naturally contains "self-referencing causal cycles"—repeated phrases or structures that act as internal hyperlinks. When a model encounters a phrase it has processed before, its attention mechanism jumps back to the earlier occurrence, bypassing the "reversal curse" and retrieving backward-facing knowledge.

Where data does not naturally provide these links, external knowledge bases can simulate them. Wang and Tam (Suffix Retrieval-Augmented Language Modeling) show that retrieving "future" suffixes from a massive data store and injecting them into the current hidden state allows a causal decoder to attend to plausible future continuations, effectively bridging the gap between unidirectional generation and bidirectional understanding.

Summary

The inherent unidirectionality of decoder-only architectures is not a fundamental limitation of what the model can compute, but rather a data-flow constraint. By repeating sequences in prompts, passing summary tokens across layers, or re-encoding intermediate states, these methods allow causal models to look into the "future" of a sequence by turning it into the past.

Key Foundational Work

Improving Input-label Mapping with Demonstration Replay for In-context Learning (2023) — contributed 1 concept elements to this direction.
MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators (2022) — contributed 1 concept elements to this direction.
Bidirectional Language Models Are Also Few-shot Learners (2023) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bidirectional_context_simulation_28489015,
  title = {bidirectional context simulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-context-simulation/rdr_284890159d3341a0b576996f655d34f5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_bidirectional_context_simulation_28489015_v1,
  title = {bidirectional context simulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-context-simulation/rdr_284890159d3341a0b576996f655d34f5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake