Asymmetric Actor-Critic Design for Stable Reinforcement Learning

This direction is merged into another direction: forward transfer deficiency. Its last version stays readable and citable here.

Version 1 of 18 papers18 conceptsfrozen 2026-10-04

In standard Actor-Critic reinforcement learning (RL), we often treat the two components as two sides of the same coin, frequently giving them identical architectures and regularization schemes. However, recent research suggests that the actor (the policy) and the critic (the value function) have fundamentally different mathematical needs.

By decoupling their structures and regularization—a concept known as functional asymmetry—researchers have unlocked significant improvements in how agents learn new tasks and generalize from offline data.

The Divergent Roles of Actor and Critic

The actor and critic serve distinct purposes. The actor must map states to actions, requiring enough flexibility to explore complex action spaces. The critic, meanwhile, must solve the Bellman equation, estimating the long-term value of those actions.

In FANS: A Flatness-Aware Network Structure for Generalization in Offline Reinforcement Learning, Da Wang, Yi Ma, and colleagues demonstrate that architectural features like residual connections and layer normalization are highly beneficial for the critic. These components guide the critic toward "flatter" loss landscapes, which improves generalization in offline settings.

Crucially, however, they found that applying these same structures to the actor can be detrimental. While residual blocks stabilize value estimation, they can inadvertently restrict the actor's action space, preventing the policy from reaching optimal solutions. This suggests that the actor and critic require distinct structural priors to function effectively.

The Plasticity Paradox in Continual Learning

The need for asymmetry becomes even more apparent in Continual Reinforcement Learning, where an agent must learn a sequence of tasks without forgetting old ones.

In Continual World, Maciej Wolczyk, Michał Zając, and Razvan Pascanu observed a "plasticity paradox." To prevent catastrophic forgetting, standard continual learning methods apply weight regularization to the entire network. While this keeps the policy stable, it cripples the critic.

The critic's job is to minimize Bellman error, but value targets are non-stationary—they change as the agent learns and as the environment's reward scales shift. If the critic is regularized (forced to stay close to old weights), it cannot adapt to the distinct reward scales of a new task. This failure to adapt prevents forward transfer, the ability to use past knowledge to learn new tasks faster.

Selective Critic Regularization

To solve this, researchers proposed selective critic regularization. The intuition is simple: stabilize the actor to prevent forgetting, but keep the critic fully plastic.

In Disentangling Transfer in Continual Reinforcement Learning, Piotr Milos and Łukasz Kuciński deconstructed the Soft Actor-Critic (SAC) algorithm and identified the critic as the primary engine of transfer. They introduced ClonEx-SAC, which applies behavioral cloning-style regularization to the actor while allowing the critic to evolve freely.

This asymmetric approach works because:

  1. The Actor provides stability: By regularizing the policy, the agent retains the "how-to" of previous tasks.
  2. The Critic provides adaptability: By remaining unconstrained, the critic can rapidly adjust to new reward dynamics and temporal dependencies without historical bias.

Why Asymmetry Works

The success of these asymmetric designs stems from acknowledging that the critic is essentially solving a moving-target regression problem, while the actor is performing a search in action space.

In offline RL, the asymmetry in FANS ensures stable value estimation through specialized architecture without over-constraining the policy's search. In continual RL, the asymmetry in ClonEx-SAC ensures that the agent can "forget" old value scales to "learn" new ones, all while the policy remains anchored to its previous successes.

By treating the actor and critic as functionally distinct entities rather than identical twins, RL agents become both more stable and more capable of transferring knowledge across their lifetimes.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_asymmetric_actor_critic_design_for_stabl_0c6e5fca,
  title = {Asymmetric Actor-Critic Design for Stable Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/asymmetric-actor-critic-design-for-stable-reinforcement-learning/rdr_0c6e5fcafc69422aa3a90b86c859b8a0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_asymmetric_actor_critic_design_for_stabl_0c6e5fca_v1,
  title = {Asymmetric Actor-Critic Design for Stable Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/asymmetric-actor-critic-design-for-stable-reinforcement-learning/rdr_0c6e5fcafc69422aa3a90b86c859b8a0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}