Generalizing Innovations Across Transformer Architectures

41 concepts54 papers1 proposal

When a machine learning breakthrough occurs—whether it is a more stable optimizer, a novel pre-training objective, or a memory-saving trick—it is rarely evaluated across every flavor of neural network. Instead, researchers naturally validate their ideas within the dominant architecture of their specific subfield. Generative large language model (LLM) researchers test on causal decoder-only models; translation and structural modeling researchers often default to encoder-decoder or encoder-only frameworks.

This practice creates an invisible boundary in machine learning research: architectural scope limitation. A mechanism that fundamentally alters or improves one structural paradigm often remains entirely unverified, or fails to transfer cleanly, when applied to another. Understanding why these innovations are bound by architecture requires looking closely at how structural design dictates internal representations, optimization dynamics, and memory layouts.


How Attention Masking Alters Internal Representations

The most fundamental divide in Transformer architectures is how tokens are allowed to look at one another. Encoder-only models utilize bidirectional attention, allowing every token to build context from both past and future tokens. Decoder-only models employ causal masking, forcing tokens to look only backward.

This structural difference completely changes the geometric properties of the model's hidden states and attention maps. For example, Talebpour et al. demonstrated in Topics in Contextualised Attention Embeddings that the bidirectional attention layers of encoder-only models implicitly encode probabilistic topic distributions, clustering words thematically in higher layers. However, because causal decoders lack future context, their attention dynamics behave entirely differently. As shown by Ma et al. in From Attention to Activation: Unravelling the Enigmas of Large Language Models, autoregressive decoder-only models develop severe attention artifacts, such as an unnatural dominance of the very first token and massive outlier activations in hidden states.

Because the underlying representation spaces are so distinct, an innovation designed to exploit or fix one paradigm rarely transfers out-of-the-box to another:

  • Bias Mitigation: The "Bias Vector" subtraction method proposed by Shirafuji et al. in Bias Vector: Mitigating Biases in Language Models with Task Arithmetic Approach successfully mitigates social biases in the parameter spaces of bidirectional encoder models, but its efficacy remains unverified on autoregressive models like GPT-2.
  • Pre-training Objectives: The self-critic pre-training framework (SCRIPT) introduced by Nijkamp et al. in SCRIPT: Self-Critic PreTraining of Transformers allows a model to act as its own discriminator. While highly effective for encoder-only architectures, adapting these discriminative objectives to generative, decoder-only models remains an open challenge.
  • RAG Calibration: In retrieval-augmented generation, the calibration techniques developed by Zhang et al. in Lexical Diversity-aware Relevance Assessment for Retrieval-Augmented Generation rely heavily on the next-token prediction mechanics of autoregressive decoders, leaving their utility for non-autoregressive or encoder-only retrieval systems unknown.

Optimization and Scaling Disconnects

The mathematical behavior of an optimizer or a scaling law is not invariant to architecture. The way gradients flow through a purely causal chain of tokens differs significantly from how they flow through an encoder-decoder bottleneck.

Consider how we understand optimization stability. When Rosie Zhao et al. deconstructed what makes an optimizer successful in Deconstructing What Makes a Good Optimizer for Autoregressive Language Models, the authors found that Adam's success in these models largely stems from its specific ability to provide adaptivity to the final layer and LayerNorm parameters. However, because this analysis was exclusively tailored to autoregressive decoder-only models, these insights into optimizer necessity do not necessarily generalize to encoder-decoder or non-transformer architectures.

Similarly, data scaling and overfitting dynamics diverge across structural boundaries. In To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis, Xue et al. noted that while the challenges of multi-epoch data repetition are thoroughly documented for encoder-decoder models, empirical evidence showing how these degradation patterns scale in massive decoder-only models is still lacking. Even downscaling behaviors do not translate: when Deshpande et al. compressed language models to ultra-low parameter scales (1M–20M parameters) in Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale, they proved that pre-training benefits persist for encoders, but whether generative decoders maintain stable scaling laws at this tiny scale remains an open question.


Hardware and Memory Layout Constraints

Modern efficiency innovations are often co-designed with the specific memory footprints of an architecture. If an optimization targets a memory bottleneck unique to one paradigm, it becomes structurally incompatible with others.

The Key-Value (KV) cache is a prime example. In a decoder-only model, generating text requires caching past token keys and values to avoid redundant computations. To alleviate this massive memory footprint, Filippova et al. introduced Random Cross-Layer Attention in Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing to share KV caches across layers. Because encoder-decoder models process the prompt and generation in separate stages, the applicability of such depth-wise cache sharing mechanisms to encoder-decoder frameworks remains unverified.

Decoder-Only Memory Optimization (e.g., QST, Stochastic KV Routing)
[Causal Prompt Processing] ---> [KV Cache Generation / Side Network Tuning] 
                                       |
                                       +---> Native to Autoregressive State

Encoder-Decoder Memory Optimization (e.g., MoWE, AdaWAC)
[Bidirectional Encoder] === Bottleneck ===> [Causal Decoder Generation]
         |                                           |
         +---> Cross-Attention / Routing Layer ------+

The same paradigm-locking applies to parameter-efficient fine-tuning (PEFT). The Quantized Side Tuning (QST) framework, proposed by Dan Zhao et al. in Quantized Side Tuning: Fast and Memory-Efficient Tuning of Quantized Large Language Models, achieves massive memory savings by training a lightweight "side network" alongside a frozen, quantized LLM. However, this framework is explicitly tailored to decoder-only blocks; its effectiveness on encoder-decoder models like T5 has not been explored. Conversely, architectural enhancements like the Mixture-of-Word-Experts (MoWE) framework developed by dos Santos et al. in Memory Augmented Language Models through Mixture of Word Experts utilize sparse routing mechanisms that have only been validated within encoder-decoder architectures.


The Task Generalization Gap

When evaluating downstream task performance, the choice of architecture can inadvertently bias the conclusions of a study. A framework that appears highly effective on an encoder-decoder model may underperform or scale poorly on a decoder-only counterpart.

For instance, in the domain of code intelligence, To et al. demonstrated in Better Language Models of Code through Self-Improvement that fine-tuned encoder-decoder models can successfully generate their own augmented data to self-improve. Yet, as the authors acknowledge, it remains an open question whether these self-improvement benefits generalize to the massive decoder-only models that dominate modern software engineering tasks. This gap is echoed by Sheikhaei et al. in An Empirical Study on the Effectiveness of Large Language Models for SATD Identification and Classification, where evaluations of technical debt classification were restricted to T5 and BERT families, leaving the capabilities of newer decoder-only models unverified.

Even when evaluating broad capabilities like zero-shot classification, empirical findings can be siloed. As Lepagnol et al. pointed out in Small Language Models Are Good Too: An Empirical Study of Zero-Shot Classification, studies comparing model scales often restrict their evaluations to causal decoder-only and encoder-decoder models, entirely excluding encoder-only architectures and thereby limiting the generalizability of their architectural insights.

Moving Beyond the Silos

The recurring pattern across modern deep learning literature is clear: an innovation is rarely "universal." True architectural generalization requires researchers to deliberately step outside the comfort zone of a single paradigm. To prove that a mechanism is a fundamental advancement in deep learning—rather than a clever exploit of a specific attention mask—it must be tested across the invisible walls that separate encoders, decoders, and encoder-decoder systems.

Key Foundational Work

SCRIPT: Self-Critic PreTraining of Transformers (2021) — contributed 2 concept elements to this direction.
Multi-Unit Transformers for Neural Machine Translation (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_architectural_scope_limitation_architect_48b52924,
  title = {architectural scope limitation (architectural scope limitations)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/architectural-scope-limitation-architectural-scope-limitations/rdr_48b5292472bf4a5b91ca3c8cad6c19c5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_architectural_scope_limitation_architect_48b52924_v1,
  title = {architectural scope limitation (architectural scope limitations)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/architectural-scope-limitation-architectural-scope-limitations/rdr_48b5292472bf4a5b91ca3c8cad6c19c5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake