Introduction
In multi-goal reinforcement learning (RL), agents must learn a value function $Q(s, a, g)$ that predicts the utility of an action $a$ in state $s$ relative to a specific goal $g$. Standard "monolithic" architectures typically concatenate these inputs into a single neural network. However, this approach often struggles with combinatorial generalization—the ability to infer the value of novel state-action-goal combinations from previously seen data. Consequently, monolithic models require extensive training data and often fail to generalize to goals outside their immediate training distribution.
Methodology: The Bilinear Decomposition
To address these limitations, Bilinear Value Networks (BVN) introduce a structural inductive bias by factorizing the Q-function into a dot product of two distinct vector fields:
$$Q(s, a, g) = f(s, a)^T \phi(s, g)$$
This decomposition separates the problem into two distinct components:
- The Local Field ($f(s, a)$): This network processes the current state and action to represent environment dynamics—essentially "what happens next" independent of the ultimate goal.
- The Global Field ($\phi(s, g)$): This network processes the state and the goal to represent "where to go next," defining the global relationship between the current position and the target.
The Q-value is maximized when the action vector $f$ aligns with the goal direction vector $\phi$, providing a clear geometric interpretation of the agent's decision-making process as a measure of alignment.
Key Insights and Expressivity
A critical feature of BVN is that the global field $\phi(s, g)$ is conditioned on the current state $s$. Unlike previous bilinear models that used a fixed goal representation $\phi(g)$, BVN allows the "direction to the goal" to adapt based on the agent's location. This is essential for navigating complex environments where the direct path might be blocked.
By forcing the architecture to respect this split, the model avoids re-learning the physics of the environment for every new goal. This disentanglement significantly improves data efficiency and allows for zero-shot or few-shot transfer by reusing the learned dynamics or goal-reaching components.
Experimental Evaluation
The researchers evaluated BVN across two challenging benchmark suites:
- Fetch Robot Suite: Includes tasks such as Reach, Push, Slide, and Pick & Place, as well as extensions like Drawer-Open/Close and Bin-picking.
- Shadow Hand Suite: A high-dimensional dexterous manipulation environment involving a 20-degree-of-freedom hand performing tasks like rotating blocks, eggs, and pens.
Key Results
- Superior Data Efficiency: BVN consistently outperformed monolithic baselines, achieving higher success rates with significantly less data in 5 out of 8 Fetch tasks and 6 out of 8 Shadow Hand tasks.
- Out-of-Distribution (OOD) Generalization: BVN demonstrated superior performance when tested on "far" goals or spatial splits (e.g., left vs. right) not seen during training, adapting much faster than monolithic networks.
- Transfer Learning: The factorization allows for efficient transfer. For instance, when moving from a "Push" task to a "Slide" task, resetting the $f$ network while keeping the $\phi$ network allowed for faster convergence.
Conclusion
Bilinear Value Networks provide a powerful architectural bias for multi-goal RL. By factorizing the Q-function into local dynamics and global intent, BVN achieves state-of-the-art data efficiency and generalization. Future work may explore sharing a single "global" network across different robot morphologies, potentially enabling a unified controller for diverse hardware platforms.
Summary updated