In multi-task and lifelong reinforcement learning (RL), agents face a fundamental trade-off between generalization and interference. Traditional approaches often rely on deep neural networks with shared backbones or hidden layers. While these architectures excel at extracting shared features, they are highly susceptible to negative interference—where learning a new task inadvertently degrades performance on a previously mastered one—and catastrophic forgetting in sequential settings.
To bypass these limitations, a powerful paradigm has emerged: factored parameter representations. Instead of sharing internal network activations, this framework shifts the sharing mechanism directly into the parameter space, decomposing complex policies into reusable functional building blocks.
The Core Intuition: Factoring the Parameter Space
The core mechanism relies on a simple yet elegant matrix factorization. Instead of optimizing an independent set of parameters $\theta_t$ for each task $t$, the parameters are represented as a linear combination of a shared latent basis (or dictionary) $L$ and a task-specific coefficient vector $s_t$:
$$\theta_t = L s_t$$
In this formulation, $L$ acts as a repository of core behavioral primitives or skills shared across the entire task distribution, while $s_t$ serves as a low-dimensional instruction vector detailing how to combine those primitives for a specific scenario.
As highlighted by Lingfeng Sun et al. in their work on Parameter-Compositional Multi-task RL (PaCo), performing interpolation directly in the parameter space decouples "what to learn" (the basis $L$) from "how to use it" (the weights $s_t$). This structural separation allows tasks to express identical, closely related, or entirely independent behaviors simply by adjusting their composition weights, effectively eliminating the negative interference common in shared-backbone networks.
Task 1 Description ──► [ s₁ ] ──┐
▼
Task 2 Description ──► [ s₂ ] ──► [ Shared Basis Dictionary L ] ──► Factored Policies (θ₁, θ₂, θ₃)
▲
Task 3 Description ──► [ s₃ ] ──┘
Efficient Lifelong Learning and Online Adaptation
When an agent encounters tasks sequentially over its lifespan, this factorization drastically accelerates learning. Instead of searching the high-dimensional space of all possible policy parameters for every new task, the agent only needs to find a sparse combination of the existing basis vectors in $L$.
This strategy forms the foundation of PG-ELLA, introduced by Haitham Bou-Ammar et al., which brought online multi-task learning to policy gradient methods. By maintaining a shared library of latent components, the agent can rapidly integrate new tasks via sparse linear combinations while simultaneously refining the shared library to improve performance on prior tasks.
To make this process safe for physical systems, Haitham Bou-Ammar et al. incorporated safety constraints into this lifelong learning framework using Bregman projections. This approach guarantees that the agent avoids unsafe states during exploration while still achieving sublinear regret. Building on these optimization principles, Jorge Mendez-Mendez et al. developed LPG-FTW, a method that directly optimizes these factored policy representations using policy gradients, ensuring efficient lifelong transfer without forgetting across highly diverse environments.
Decentralized and Communication-Efficient Collaboration
The compact nature of factored representations makes them uniquely suited for distributed multi-agent networks. Because a policy is fully characterized by its low-dimensional coefficient vector $s_t$ and the shared dictionary $L$, agents can collaborate and pool knowledge without the massive communication overhead of transmitting raw data or full model gradients.
For instance, Salam El Bsat et al. formulated multi-task policy gradients as a decentralized consensus optimization problem, achieving linear convergence speeds. Similarly, the Collective Lifelong Learning Algorithm (CoLLA), developed by Mohammad Rostami et al. and extended in Mohammad Rostami et al., enables a network of independent agents to collaboratively learn and refine a shared knowledge dictionary. This decentralized dictionary learning minimizes communication costs and preserves data privacy, making it highly effective for resource-constrained applications like Internet of Things (IoT) networks (Zhenzhen Gong et al.).
Beyond Forward Policies: Rewards and Zero-Shot Transfer
The utility of parameter composition extends beyond forward policy parameters; the same mathematical intuition applies to reward functions and zero-shot generalization.
In Inverse Reinforcement Learning (IRL), where an agent must deduce an underlying reward function from human demonstrations, Jorge A. Mendez et al. demonstrated that reward parameters can also be factorized into a shared latent basis and sparse task-specific coefficients. This allows the agent to extract reusable behavioral primitives from demonstrations, significantly reducing the number of samples needed to understand a new task.
Furthermore, this shared latent space can be mapped directly to external metadata to enable zero-shot transfer. As shown by Mohammad Rostami et al., high-level task descriptions (such as textual features or physical parameters) can be factorized using a separate dictionary but forced to share the exact same sparse coefficients $s_t$ as the policy parameters. This coupling creates a shared latent bridge:
$$\text{Task Description} \approx D s_t \iff \text{Policy Parameters} \approx L s_t$$
When a completely unseen task arrives, the agent can instantly infer the optimal composition coefficients $s_t$ from the task description dictionary $D$, and project them through $L$ to construct an effective policy immediately, achieving zero-shot transfer without requiring initial environmental interaction.
Key Foundational Work
Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regret (2015) — contributed 2 concept elements to this direction.
Online Multi-Task Learning for Policy Gradient Methods (2014) — contributed 1 concept elements to this direction.