When training machine learning models across multiple domains, tasks, or decentralized clients, we inevitably encounter a fundamental trade-off. If we train a single global model on all data combined, we risk washing out the critical nuances of individual tasks—a phenomenon known as negative transfer. Conversely, if we isolate each task and train separate models independently, we suffer from data sparsity, failing to leverage common underlying structures.
Parameter decomposition breaks this zero-sum game. Instead of treating a model's weights as a monolithic block, this research direction mathematically splits parameters into distinct components: one that captures universal, shared knowledge, and another that isolates task-specific variations or noise. By structuring the parameter space this way, models can simultaneously maximize cross-task generalization and local specialization.
Additive Superposition: Separating Signal from Noise
The most intuitive way to decouple shared and specific knowledge is through an additive superposition. For any given task $t$, the predictive parameters $W_t$ are modeled as:
$$W_t = W_{\text{shared}} + \Delta W_t$$
Here, $W_{\text{shared}}$ acts as a global anchor learned across all tasks, while $\Delta W_t$ represents a localized deviation or tweak tailored to the unique characteristics of task $t$.
Monolithic Weight (W) Decomposed Weights
[ Total ] = [ Shared ] + [ Specific ]
(Prone to noise) (Universal) (Task Tweak)
This formulation provides a clean mathematical filter. The shared matrix extracts the common signal, while the task-specific matrix catches the residual variations. As surveyed by Xiaokang Liu et al. in Multitask Learning 1997–2024: Part II Regularization and Optimization, splitting the parameter matrix into a shared component and a task-specific outlier component prevents unique task variations from corrupting the generalizable representation.
This mechanism is highly effective when dealing with messy, real-world data. Ali Jalali et al. demonstrate in A Dirty Model for Multi-task Learning that modeling parameters as a superposition of two distinct components allows a system to represent different structural properties simultaneously. For instance, in spatial-temporal forecasting, Xiangyu Zhao et al. utilize this separation in Multi-Type Urban Crime Prediction to ensure that common environmental signals (like foot traffic) are captured globally without drowning out the distinct patterns associated with specific crime types.
Furthermore, this additive structure can be extended hierarchically. Lei Han et al. show in Learning Multi-Level Task Groups in Multi-Task Learning that task parameters can be represented as an additive decomposition of multiple component matrices, where each matrix captures relationships at a distinct level of granularity. This allows the model to represent nested similarities by summing broad shared features with fine-grained, group-specific adjustments.
Multiplicative Factorization: Latent Task Bases
Instead of adding a deviation to a global baseline, we can factorize the weight matrix multiplicatively to discover a low-dimensional latent space of tasks:
$$W = B \cdot C$$
where $B$ is a matrix containing shared "basis" prototypes or latent features, and $C$ is a coefficient matrix that dictates how much each individual task relies on those bases.
Task Weights (W) Basis Matrix (B) Coefficient Matrix (C)
[ Task 1 ] [ Basis 1 ] [ c11 c12 ]
[ Task 2 ] = [ Basis 2 ] × [ c21 c22 ]
[ Task 3 ] [ Basis 3 ] [ c31 c32 ]
This perspective treats tasks as combinations of fundamental building blocks. Abhishek Kumar formalized this concept in Learning Task Grouping and Overlap in Multi-task Learning, demonstrating that modeling task parameters as sparse combinations of latent basis tasks allows for a "continuum" of sharing. Rather than forcing all tasks to share equally, each task selectively reconstructs its parameter vector from the shared basis pool.
This principle translates directly to computer vision and multi-domain learning. In Learning to Share Latent Tasks for Action Recognition, Qiang Zhou et al. decompose a classifier matrix into a dictionary of latent task representations and a sparse matrix of combination weights. This allows complex action categories to be represented as combinations of shared, fundamental motion patterns (like jumping or swinging) rather than independent silos.
Multiplicative decomposition also serves as a powerful tool for parameter efficiency and model compression. In Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression, Jingcun Wang et al. approximate weights across different layers of Large Language Models as a product of a common basis matrix and layer-specific coefficient matrices. Because the core transformation vectors are shared globally across layers, the model preserves unique layer identities through tiny coefficient adjustments, enabling high compression ratios without sacrificing performance.
Preventing Interference in Continual and Federated Learning
The utility of parameter decomposition extends far beyond static multi-task learning; it serves as a critical defense mechanism against interference in dynamic environments, such as continual learning and federated learning.
In continual learning, models must learn a sequence of tasks over time. The primary challenge is catastrophic forgetting: updating weights for a new task typically overwrites what was learned for previous ones. To solve this, Jaehong Yoon et al. introduced Additive Parameter Decomposition (APD) in Scalable and Order-robust Continual Learning with Additive Parameter Decomposition. By decomposing network weights into a global shared matrix, a task-specific attention mask, and a sparse residual component, the model isolates task-specific updates from the shared backbone, effectively eliminating forgetting.
Continual Learning Update:
Shared Matrix [======] --> Frozen or retroactively updated
Task Mask [ X ] --> Isolates gradient updates
Sparse Tweak [ + ] --> Captures task-specific idiosyncrasies
In Personalized Federated Learning, decentralized clients must collaborate to train a global model while maintaining local customization. Standard federated averaging fails when clients have highly heterogeneous data distributions. The authors address this in Federated Continual Learning with Weighted Inter-client Transfer (FedWeIT) by partitioning parameters into global shared weights, local base weights, and sparse task-adaptive components. This allows clients to share foundational knowledge via the global weights while protecting their local variations from being corrupted during aggregation.
Similarly, Hao Su shows in Decoupling General and Personalized Knowledge in Federated Learning via Additive and Low-Rank Decomposition that decomposing weight matrices into an additive sum of a shared full-rank component and a private low-rank component allows every single parameter in the network to simultaneously contribute to both global generalization and local personalization. This avoids the rigid boundaries of traditional layer-by-layer partitioning, providing a more surgical way to separate general knowledge from client-specific noise.
Key Foundational Work
Transferring Multi-device Localization Models using Latent Multi-task Learning (2008) — contributed 1 concept elements to this direction.
Learning from Relevant Tasks Only (2007) — contributed 1 concept elements to this direction.