Introduction
In policy-based reinforcement learning (RL), agents learn to map states to action distributions. While deep neural networks (NNs) are the standard choice for these mappings, they are often criticized as "black boxes" that are difficult to tune, prone to convergence issues, and frequently over-parameterized. Crucially, NNs often fail to explicitly exploit the inherent structure of many RL problems, such as state redundancy—where similar states should naturally lead to similar actions.
This paper proposes a shift from neural architectures to low-rank tensors. By arranging policy parameters into a multi-dimensional table and enforcing a low-rank structure via PARAFAC decomposition, the authors introduce a family of "Tensor Low-Rank" (TLR) algorithms. These methods are designed to be more parameter-efficient and faster to converge than standard deep RL approaches.
Methodology
The framework for Multilinear Tensor Low-Rank Approximation involves three primary stages:
- Tensorization: Policy parameters (such as the means of a Gaussian distribution for continuous control) are arranged into a high-dimensional tensor $\Theta$.
- PARAFAC Decomposition: To combat the "curse of dimensionality," the giant tensor is decomposed using the PARAFAC (or CP) model. This represents the tensor as a sum of $K$ rank-1 tensors, reducing parameter growth from exponential to linear relative to the number of state dimensions.
- Algorithm Integration: The authors derive closed-form policy scores (gradients) for these models, allowing them to be integrated into standard RL algorithm families, including REINFORCE (TLRPG), Actor-Critic (TLRAC), TRPO (TRTLRPO), and PPO (PTLRPO).
Unified Training Loop
The integration of tensor models into policy-gradient methods follows a structured iterative process. As shown in the algorithmic framework below, the process alternates between sampling trajectories and updating the actor (the tensor-based policy) and the critic.
Experimental Evaluation
The researchers evaluated TLR methods across two distinct domains:
- Classical Control: Testing on OpenAI Gym's
MountainCarandPendulumenvironments in both discrete and continuous variants. - Wireless Communications: A complex 6D state-space simulation of an opportunistic multiple-access system, requiring the agent to manage battery levels and data queues across multiple channels.
Key Insights and Results
Policies are Naturally Low-Rank
A striking finding of this research is that even when standard NNs are used, the resulting policies are implicitly low-rank. The authors demonstrated that NN-trained policies can be compressed into low-rank tensors with minimal error, justifying the use of tensor models from the outset.
Performance and Efficiency
- Superior Convergence: TLR methods reached peak rewards significantly faster than NN counterparts. In the
Pendulumenvironment, TLR policies achieved stability in roughly half the episodes required by NNs. - Parameter Efficiency: TLR models achieved high performance with fewer parameters. In the wireless communication task, the PTLRPO model used approximately 900 parameters compared to 1100 for the NN, while reaching a steady state much more rapidly.
- Theoretical Reliability: The paper provides a formal proof (Theorem 1) ensuring that the Tensor Low-Rank Policy Gradient algorithm converges to a stationary point, offering mathematical guarantees often absent in deep RL.
Conclusion and Future Directions
While the TLR approach offers significant gains in efficiency and convergence, some challenges remain. The current formulation relies on discretization or basis functions for continuous state spaces. Future work may explore "functional tensors" or Tensor Trains to handle high-dimensional continuous spaces more natively. Additionally, automating the selection of the rank $K$—currently a hyperparameter—would further enhance the usability of these models in complex environments like pixel-based Atari games, which currently require separate feature extractors.
Summary updated