Bi-linear Value Networks for Multi-goal Reinforcement Learning

Zhang-Wei Hong, Ge Yang, Pulkit AgrawalICLR 2022 · OpenReview

Introduction

Universal value functions are a cornerstone of off-policy multi-goal reinforcement learning (RL). In this setting, an agent learns a policy conditioned on a specific goal $g$. The de-facto paradigm is to approximate the Q-function, $Q(s, a, g)$, using a "monolithic" neural network. This network estimates the value of taking action $a$ in state $s$ to reach goal $g$.

However, monolithic networks are notoriously data-inefficient. By treating the input $(s, a, g)$ as a single joint vector, they lack the inductive biases necessary to model the complex, compositional interactions between the state and the goal. Consequently, they struggle with combinatorial generalization—for instance, failing to reach a known goal $g_B$ from a known state $s_A$ simply because that specific combination wasn't seen during training. Bilinear Value Networks (BVN) address this by factorizing the Q-function into two disentangled components, significantly boosting data efficiency and zero-shot generalization.

Methodology: Bilinear Value Networks

To overcome the limitations of monolithic architectures, BVN introduces a specific structural constraint (an inductive bias) into the Q-function. Instead of a single large network, the Q-function is decomposed into a low-rank approximation via a dot product of two separate vector fields:

$$Q(s, a, g) = f(s, a)^T \phi(s, g)$$

Architecture comparison between monolithic and bilinear value networks, alongside a vector field visualization.
  • $f(s, a)$ (Local Dynamics): This vector field captures the environment's local dynamics at state $s$. It focuses on the immediate consequences of an action, independent of the ultimate goal ("what happens next").
  • $\phi(s, g)$ (Global Target): This vector field captures the global relationship between the current state and the goal ("where to go next").

The Q-value is maximized when the "action direction" $f(s, a)$ aligns perfectly with the "goal direction" $\phi(s, g)$.

The Importance of State-Dependent Goal Representations

While bilinear decompositions have been explored previously (e.g., in early Universal Value Function Approximators), prior versions often made the goal component $\phi$ dependent only on $g$. BVN crucially makes $\phi$ dependent on both $s$ and $g$.

Comparison of goal representations with an obstacle. Making the goal representation dependent on state allows for complex navigation.

Duplicating the state $s$ in both branches of the network is essential. As illustrated in 2D navigation examples with obstacles, an agent cannot accurately determine the "direction to the goal" without knowing its current location relative to environmental constraints. This state-dependency allows the model to represent the complex, non-linear relationships required for 3D robotic tasks while preserving the benefits of disentanglement.

Experimental Setup and Key Results

The researchers evaluated BVN on several challenging simulated environments, including the standard Fetch Robot Suite (Reach, Push, Slide, Pick & Place), a more complex Fetch-extension suite, and high-dimensional (20-DoF) dexterous manipulation tasks using a Shadow Hand.

Key Findings: * Superior Sample Efficiency: BVN achieved higher success rates using significantly less data than monolithic baselines in 5 out of 8 Fetch tasks and 6 out of 8 Shadow Hand tasks. * Combinatorial Generalization: In "Near-to-Far" tests—where agents are trained on nearby goals and tested on distant ones—BVN adapted much faster during fine-tuning than standard networks. * Robustness to Latent Size: BVN outperformed the monolithic baseline even with a very small latent dimension (e.g., $d=3$), though performance continued to scale up to $d=16$. * Transfer Learning Potential: Because $f(s, a)$ is theoretically goal-agnostic, it represents the "physics" of the robot. The authors demonstrated that by freezing $\phi$ and resetting $f$, they could improve transfer between tasks with similar goals but different dynamics (e.g., transferring from "Pushing" to "Sliding").

Limitations and Future Work

While BVN provides a strong inductive bias for disentanglement, the networks do not always achieve perfect separation in practice. Experiments showed that fine-tuning the theoretically "goal-agnostic" $f$ component still provided slight improvements when adapting to new goals, suggesting some goal information inevitably leaks in.

Additionally, certain variants of the method (such as using L2 distance instead of a dot product) require rewards to be strictly non-positive or non-negative, limiting flexibility.

Looking forward, a promising direction is leveraging BVN for transfer across different robot morphologies. For example, one could train a single global target function $\phi(s, g)$ for a specific task, and then learn different local dynamics functions $f(s, a)$ for entirely different robotic arms or hands.

Summary updated

Abstract

Universal value functions are a core component of off-policy multi-goal reinforcement learning. The de-facto paradigm is to approximate Q(s, a, g) using monolithic neural networks which lack inductive biases to produce complex interactions between the state s and the goal g. In this work, we propose a bilinear decomposition that represents the Q-value via a low-rank approximation in the form of a dot product between two vector fields. The first vector field, f(s, a), captures the environment's local dynamics at the state s; whereas the second component, ϕ(s, g), captures the global relationship between the current state and the goal. We show that our bilinear decomposition scheme improves sample efficiency over the original monolithic value approximators, and transfer better to unseen goals. We demonstrate significant learning speed-up over a variety of tasks on a simulated robot arm, and the challenging task of dexterous manipulation with a Shadow hand.
BibTeX
@inproceedings{hong2022bi,
  author = {Zhang-Wei Hong and Ge Yang and Pulkit Agrawal},
  title = {{Bi-linear Value Networks for Multi-goal Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=LedObtLmCjS},
}
Figure