Shared Encoders for Multi-Task Representation Learning

32 concepts32 papers1 proposal

When building natural language processing pipelines, a naive approach is to train an independent model for every distinct task. If an application needs to classify sentiment, identify offensive language, and extract named entities, a standard deployment might run three separate large language models. This strategy is computationally extravagant and ignores a fundamental truth: these tasks rely on the same underlying linguistic structures.

Instead of maintaining isolated models, modern architectures favor hard parameter sharing via a shared-encoder multi-task framework. In this setup, a single deep neural network backbone (the encoder) processes the input text to generate a unified contextual representation. From this shared foundation, multiple lightweight, task-specific "heads" branch out to make individual predictions.

This approach works remarkably well across diverse NLP domains, driven by three core mechanisms: geometric regularization, cross-task synergy, and computational consolidation.

Regularization Through Diverse Inductive Biases

Why does a shared encoder often outperform a single-task model on its own objective? The answer lies in how multi-task training regularizes the model's internal representation space.

When a model trains on a single objective, it risks overfitting to the superficial statistical patterns or noise inherent to that specific dataset. By contrast, forcing a single encoder to simultaneously solve diverse tasks alters the geometry of the embedding space. As Subramanian et al. demonstrate in Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning, combining the unique inductive biases of tasks like translation, parsing, and logical inference forces the encoder to find a highly generalized representation. The backpropagation of gradients from completely different loss functions acts as a powerful constraint, preventing the network from tuning itself too narrowly.

                  ┌──► Task A Head (e.g., Translation Loss)
                  │
Raw Text ──► Shared Encoder ──► Task B Head (e.g., Parsing Loss)
                  │
                  └──► Task C Head (e.g., Inference Loss)

This branching structure allows low-level linguistic features—such as syntax and core semantics—to be shared across primary and auxiliary tasks, while keeping specialized decision logic isolated in the output layers, as shown by Cohan et al. in Structural Scaffolds for Citation Intent Classification in Scientific Publications. Similarly, Zhang et al. note in Multi-Label Transfer Learning for Multi-Relational Semantic Similarity that aggregating losses from multiple semantic relations in a single training pass allows the shared backbone to capture subtle linguistic nuances that single-task models miss entirely.

Cross-Task Synergy and Mutual Reinforcement

Beyond general regularization, tasks within a shared encoder often act as cooperative signals, where learning one task directly illuminates another.

Consider the relationship between identifying entities and extracting the relations between them. Chenglong Zhang demonstrates in Position-aware Joint Entity and Relation Extraction with Attention Mechanism that a single transformer backbone processing both tasks allows for the mutual reinforcement of entity and relation signals. Knowing that a text span represents a specific entity type constrains the plausible relations it can hold, and this structural dependency is natively captured within the shared encoder's attention layers.

This synergy is particularly potent when pairing a complex target task with an intuitive auxiliary task. For instance, Rodríguez et al. reveal in Paper Bullets: Modeling Propaganda with the Help of Metaphor that using metaphor detection as an auxiliary task helps a shared encoder capture the non-literal, persuasive rhetoric essential for identifying propaganda. In the domain of social media moderation, Park et al. show in "Why do I feel offended?" - Korean Dataset for Offensive Language Identification that a shared encoder leveraging representations from sentiment and profanity detection significantly improves a model's grasp of fine-grained, implicit offensive language.

Furthermore, this shared grounding aids in domain adaptation. Dukic et al. demonstrate in Leveraging Open Information Extraction for More Robust Domain Transfer of Event Trigger Detection that training a shared encoder on both specialized event detection and general Open Information Extraction (OIE) anchors the model to universal subject-relation-object structures. This structural anchor prevents the performance drops typically seen when moving a model across different text domains.

Consolidating Computational Overhead

While the statistical benefits of multi-task learning are compelling, the practical engineering advantages are equally profound. Deploying independent large language models for every linguistic sub-task scales linearly in memory and compute costs—creating a massive bottleneck for production systems.

Shared-encoder architectures solve this by consolidating the heavy lifting into a single forward pass. As Che et al. demonstrate with the N-LTP platform in N-LTP: An Open-source Neural Language Technology Platform for Chinese, a single shared transformer encoder can extract features for six distinct core NLP tasks simultaneously, eliminating computational redundancy and slashing memory overhead.

In specialized domains like biomedicine, where models must parse dense literature for numerous distinct concepts, this efficiency is critical. Jeong et al. show in BERN2: an advanced neural biomedical named entity recognition and normalization tool that using a single pre-trained language model backbone with lightweight, task-specific output layers allows for the parallel processing of nine different entity types. This architectural choice makes it possible to run large-scale models efficiently on a single hardware accelerator.

Even in complex evaluation frameworks, this consolidation holds true. Sujan Reddy A shows in Automating Human Evaluation of Dialogue Systems that branching multiple evaluation heads from a single pooled transformer representation yields highly accurate, human-like judgments across naturalness, quality, and informativeness. By avoiding the need to train independent models for each metric, the shared architecture achieves high performance at a fraction of the computational cost. Similarly, Wang et al. apply this principle to knowledge-base question answering in Retrieval, Re-ranking and Multi-task Learning for Knowledge-Base Question Answering, using lightweight layers over a shared BERT encoder to minimize total parameter counts while enabling cross-task feature learning.

Key Foundational Work

Structural Scaffolds for Citation Intent Classification in Scientific Publications (2019) — contributed 1 concept elements to this direction.
Multi-Label Transfer Learning for Multi-Relational Semantic Similarity (2019) — contributed 1 concept elements to this direction.
Multitask Parsing Across Semantic Representations (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_shared_encoder_multitask_architecture_a8c8f31d,
  title = {shared-encoder multitask architecture},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/shared-encoder-multitask-architecture/rdr_a8c8f31de2904341bba2ccd24acadc3a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_shared_encoder_multitask_architecture_a8c8f31d_v1,
  title = {shared-encoder multitask architecture},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/shared-encoder-multitask-architecture/rdr_a8c8f31de2904341bba2ccd24acadc3a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake