For years, the natural language processing community built its foundational insights on two primary architectural pillars: encoder-only models (like BERT) for discriminative tasks, and encoder-decoder models (like T5) for conditional generation. These architectures allowed researchers to develop a rich toolkit of optimization strategies, parameter-efficient tuning methods, and alignment paradigms.
However, the modern landscape has consolidated around a different paradigm: massive, autoregressive, decoder-only foundation models. This shift introduces a critical question: Do the techniques designed for bidirectional encoders or symmetric sequence-to-sequence models actually work when applied to purely causal, generative decoders? Investigating this cross-architecture transferability reveals that structural differences—such as causal masking, token-by-token generation, and massive parameter scaling—fundamentally alter how models internalize and manipulate representations.
The Mechanics of Causal Prompting and Adaptation
In encoder-decoder models, parameter-efficient fine-tuning (PEFT) methods like soft prompting or adapters operate within a bidirectional context. A soft prompt prepended to the input is processed entirely by the encoder, allowing every input token to attend to it symmetrically.
When we transition to decoder-only models, this dynamic changes. As noted by Seonghyeon Ye et al. in their work on soft prompt retrieval (Retrieval of Soft Prompt Enhances Zero-Shot Task Generalization and Efficiently Enhancing Zero-Shot Performance of Instruction Following Model via Retrieval of Soft Prompt), continuous embeddings interact differently with the autoregressive generation process. In a decoder-only model, a retrieved soft prompt becomes part of the causal history; it cannot look forward into the sequence, meaning its influence propagates strictly token-by-token.
Similarly, instance-dependent prompt tuning—where prompt tokens adapt dynamically to the input text—has shown significant success in encoder-based systems. However, Xianjun Yang et al. point out in Position Really Matters: Towards a Holistic Approach for Prompt Tuning that the behavior of these dynamic prompts remains an open question in massive decoder-only models, where the prompt's position relative to the causal generation boundary is highly sensitive.
The structural shift also impacts modular adaptation layers. Techniques like Ladder Side-Tuning (LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning by Jaemin Cho et al.) and instance-specific parameter generation (Hyperdecoders: Instance-specific decoders for multi-task NLP by Hamish Ivison et al.) bypass or modify internal transformer blocks to save memory. While highly effective in encoder-decoder frameworks, Zheng Zhao et al. observe in To Adapt or to Fine-tune: A Case Study on Abstractive Summarization that the trade-offs between parameter efficiency and performance shift dramatically as the underlying model scales up to massive decoder-only foundation sizes. Extreme compression methods, such as the element-wise transformations proposed by Yuyan Chen et al. in Hadamard Adapter: An Extreme Parameter-Efficient Adapter Tuning Method for Pre-trained Language Models, face unverified representational requirements when forced to handle the complex autoregressive states of pure decoders.
Redefining the Output Space: Classification vs. Generation
Beyond internal representations, the way an architecture maps its final hidden states to an output space dictates how it can be aligned or distilled. Encoder-only models typically append a small classification head to map a specialized token (like [CLS]) to a discrete set of class labels. Decoder-only models, conversely, map every token's hidden state to a massive vocabulary distribution to predict the next token.
This distinction creates friction when transferring head-alignment techniques. For instance, Viet Dao et al. utilize Optimal Transport in Lifelong Event Detection via Optimal Transport to align classification heads with pre-trained linguistic priors to prevent catastrophic forgetting. While this works beautifully for encoder-based models, the authors note that the relationship between the output head and the vocabulary distribution is fundamentally different in massive autoregressive decoders, where prompting replaces explicit classification heads.
Similarly, distillation strategies often rely on reusing the teacher's classifier head or intermediate layers to guide a smaller student model. As shown by Hang Zhang et al. in Intermediate Layer Distillation with the Reused Teacher Classifier: A Study on the Importance of the Classifier of Attention-based Models, reusing teacher components stabilizes training in encoder models, but its efficacy remains unverified in modern decoder-only architectures where intermediate representations serve a generative rather than discriminative purpose.
This challenge extends to preference optimization frameworks; while Chanhwi Kim et al. successfully apply Direct Preference Optimization (DPO) to encoder-decoder models for entity linking in Learning from Negative Samples in Biomedical Generative Entity Linking, the distinct attention mechanisms of pure decoders mean that preference signals must calibrate a generative trajectory rather than a static sequence-to-sequence mapping.
Linearity and Scaling in Weight Space
A final facet of cross-architecture generalization lies in the geometry of the model's weight space. In smaller, encoder-based models, adaptation patterns across different layers are often remarkably consistent. This consistency allows methods like CondLoRA (A Single Linear Layer Yields Task-Adapted Low-Rank Matrices by Hwichan Kim et al.) to generate low-rank matrices across layers using a single shared linear layer. However, whether these universal transformation patterns persist in massive decoder-only models remains an open question, as scaling laws may introduce significant layer-specific variance.
This geometric predictability also underpins "task arithmetic"—the practice of adding or subtracting fine-tuned weight vectors to steer model behavior. While Marinela Parovic et al. demonstrate in Investigating the Potential of Task Arithmetic for Cross-Lingual Transfer that weight-space arithmetic is a powerful tool for cross-lingual transfer in encoder-only models, it remains uncertain if these modular vector additions generalize effectively to generative decoders.
When merging multiple domain-specific adapters, Tuc Van Nguyen et al. show in Generalizability of Mixture of Domain-Specific Adapters from the Lens of Signed Weight Directions and its Application to Effective Model Pruning that "sign conflicts" (where weights cancel each other out) can be mitigated using sign-based merging strategies. Yet, because decoder-only models process information through a strictly causal lens, the manifestation of these weight conflicts across layers may follow entirely different mathematical dynamics.
Ultimately, scaling up parameter counts does not guarantee that a decoder-only model will inherit the capabilities of its encoder-decoder predecessors. In fact, Niklas Muennighoff et al. reveal in Crosslingual Generalization through Multitask Finetuning that increasing the parameter size of decoder-only models can sometimes result in suboptimal instruction-following performance compared to smaller encoder-decoder models. This underscores the core reality of this research direction: bridging the architectural gap requires more than just scaling up; it demands a fundamental re-evaluation of how optimization, alignment, and adaptation techniques interact with the unique mechanics of the causal decoder.
Key Foundational Work
Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments (2022) — contributed 1 concept elements to this direction.
Maximal Multiverse Learning for Promoting Cross-Task Generalization of Fine-Tuned Language Models (2021) — contributed 1 concept elements to this direction.
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning (2022) — contributed 1 concept elements to this direction.