Bilinear value networks

Zhang-Wei Hong, Ge Yang, Pulkit AgrawalarXiv 2022 · DOI · arXiv

Introduction

In multi-goal reinforcement learning (RL), agents must learn a value function $Q(s, a, g)$ that predicts the utility of an action $a$ in state $s$ relative to a specific goal $g$. Standard "monolithic" architectures typically concatenate these inputs into a single neural network. However, this approach often struggles with combinatorial generalization—the ability to infer the value of novel state-action-goal combinations from previously seen data. Consequently, monolithic models require extensive training data and often fail to generalize to goals outside their immediate training distribution.

Methodology: The Bilinear Decomposition

To address these limitations, Bilinear Value Networks (BVN) introduce a structural inductive bias by factorizing the Q-function into a dot product of two distinct vector fields:

$$Q(s, a, g) = f(s, a)^T \phi(s, g)$$

Comparison of monolithic and bilinear architectures. The bilinear approach decomposes the Q-function into local dynamics $f(s, a)$ and global intent $\phi(s, g)$.

This decomposition separates the problem into two distinct components:

  1. The Local Field ($f(s, a)$): This network processes the current state and action to represent environment dynamics—essentially "what happens next" independent of the ultimate goal.
  2. The Global Field ($\phi(s, g)$): This network processes the state and the goal to represent "where to go next," defining the global relationship between the current position and the target.

The Q-value is maximized when the action vector $f$ aligns with the goal direction vector $\phi$, providing a clear geometric interpretation of the agent's decision-making process as a measure of alignment.

Key Insights and Expressivity

A critical feature of BVN is that the global field $\phi(s, g)$ is conditioned on the current state $s$. Unlike previous bilinear models that used a fixed goal representation $\phi(g)$, BVN allows the "direction to the goal" to adapt based on the agent's location. This is essential for navigating complex environments where the direct path might be blocked.

State-conditioned goal vectors allow the agent to navigate around obstacles by adjusting the direction of $\phi(s, g)$.

By forcing the architecture to respect this split, the model avoids re-learning the physics of the environment for every new goal. This disentanglement significantly improves data efficiency and allows for zero-shot or few-shot transfer by reusing the learned dynamics or goal-reaching components.

Experimental Evaluation

The researchers evaluated BVN across two challenging benchmark suites:

  • Fetch Robot Suite: Includes tasks such as Reach, Push, Slide, and Pick & Place, as well as extensions like Drawer-Open/Close and Bin-picking.
  • Shadow Hand Suite: A high-dimensional dexterous manipulation environment involving a 20-degree-of-freedom hand performing tasks like rotating blocks, eggs, and pens.

Key Results

  • Superior Data Efficiency: BVN consistently outperformed monolithic baselines, achieving higher success rates with significantly less data in 5 out of 8 Fetch tasks and 6 out of 8 Shadow Hand tasks.
  • Out-of-Distribution (OOD) Generalization: BVN demonstrated superior performance when tested on "far" goals or spatial splits (e.g., left vs. right) not seen during training, adapting much faster than monolithic networks.
  • Transfer Learning: The factorization allows for efficient transfer. For instance, when moving from a "Push" task to a "Slide" task, resetting the $f$ network while keeping the $\phi$ network allowed for faster convergence.

Conclusion

Bilinear Value Networks provide a powerful architectural bias for multi-goal RL. By factorizing the Q-function into local dynamics and global intent, BVN achieves state-of-the-art data efficiency and generalization. Future work may explore sharing a single "global" network across different robot morphologies, potentially enabling a unified controller for diverse hardware platforms.

Summary updated

Abstract

The dominant framework for off-policy multi-goal reinforcement learning involves estimating goal conditioned Q-value function. When learning to achieve multiple goals, data efficiency is intimately connected with the generalization of the Q-function to new goals. The de-facto paradigm is to approximate Q(s, a, g) using monolithic neural networks. To improve the generalization of the Q-function, we propose a bilinear decomposition that represents the Q-value via a low-rank approximation in the form of a dot product between two vector fields. The first vector field, f(s, a), captures the environment's local dynamics at the state s; whereas the second component, {\phi}(s, g), captures the global relationship between the current state and the goal. We show that our bilinear decomposition scheme substantially improves data efficiency, and has superior transfer to out-of-distribution goals compared to prior methods. Empirical evidence is provided on the simulated Fetch robot task-suite and dexterous manipulation with a Shadow hand.
BibTeX
@misc{hong2022bilinear,
  author = {Zhang-Wei Hong and Ge Yang and Pulkit Agrawal},
  title = {{Bilinear value networks}},
  year = {2022},
  eprint = {2204.13695},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.13695},
}
Figure