Transductive Reward Inference on Graph

Bohao Qu, Xiaofeng Cao, Qing Guo, Yi Chang, Ivor W. Tsang, Chengqi Zhang

Abstract

Offline Reinforcement Learning (RL) provides a pathway for training agents on fixed datasets without the risks associated with real-time exploration. However, the practical application of offline RL is frequently bottlenecked by the absence of reward signals in real-world datasets. To address this, we present TRAIN (Transductive RewArds INference), a framework designed to infer rewards across massive unlabelled datasets using only a small fraction of human-annotated samples. By modeling state-action pairs as nodes in a propagation graph and utilizing multi-factor decomposition, TRAIN enables stable and effective policy learning in "reward-blind" environments. Our evaluations on Meta-World and DeepMind Control Suite show that TRAIN significantly outperforms existing baselines, maintaining high performance even with only 10% of the data labeled.

Introduction

Offline Reinforcement Learning is essential for domains such as healthcare, robotics, and autonomous driving, where "trial and error" interaction is either dangerous or prohibitively expensive. While large-scale datasets of sequences (states and actions) are increasingly available, they often lack the reward labels necessary to guide an RL agent. Manually annotating every transition in a dataset containing millions of points is unfeasible.

Existing approaches to this problem typically rely on sparse binary labels (e.g., success/failure for an entire episode) or require high-quality expert demonstrations. These methods often fail to provide the granular feedback required for complex tasks. TRAIN addresses this challenge through transductive inference, propagating rewards from a sparse set of labeled nodes to the rest of the dataset by leveraging the inherent structural relationships between state-action pairs.

The TRAIN Framework

The TRAIN framework operates through a three-stage pipeline: graph construction, transductive reward propagation, and offline policy optimization.

1. Reward Propagation Graph Construction

In the TRAIN architecture, every state-action pair $(s, a)$ in the offline dataset is represented as a node in a graph. The core challenge lies in defining the edge weights—the similarity between nodes. Rather than treating states as monolithic vectors, TRAIN employs Multi-Factor Decomposition. States and actions are broken down into sub-factors, such as specific joint angles, torso height, or motor torques. A learnable reward-shaping function, $f_\Theta$, is used to tune the contribution of these factors, ensuring that the graph captures the contextual properties most relevant to the underlying reward structure.

2. Transductive Inference

Unlike inductive learning, which seeks to learn a general mapping function, transductive learning reasons directly from observed training cases to specific unlabelled instances within the same dataset. Rewards are propagated from the small set of labeled nodes to unlabelled nodes through an iterative message-passing process. The authors provide a mathematical guarantee of convergence for this process, proving that the iterative formula reaches a unique fixed point as long as the transition matrix between unlabelled nodes remains stable.

3. Policy Learning

Once the rewards for the entire dataset have been inferred, the "completed" dataset—now containing state, action, and estimated reward triplets—is used to train a policy. For this stage, TRAIN utilizes Critic-Regularized Regression (CRR), a standard offline RL algorithm that is robust to the distribution shifts common in offline data.

Experimental Evaluation

The researchers evaluated TRAIN across two primary benchmarks for continuous control:

  • Meta-World: A suite of robotic manipulation tasks including "Box Close," "Pick Place," and "Press Button."
  • DeepMind Control Suite (DMC): Locomotion tasks such as "Cheetah Run" and "Humanoid Walk."

To test the versatility of the graph-based approach, the framework was also applied to image-based variants of these tasks. In these scenarios, RGB channels and temporal frames were treated as distinct factors during the graph construction phase.

Key Results

TRAIN demonstrated substantial improvements over strong baselines, including Behavior Cloning (BC), Time-guided rewards (TGR), and Unlabelled Data Sharing (UDS):

  • Performance Gains: In the Meta-World "Box Close" task, TRAIN achieved a return of 3897, compared to 1371 for the leading baseline.
  • Label Efficiency: The system proved remarkably efficient, requiring only 10% to 15% of the data to be annotated to achieve performance comparable to training on a fully labeled dataset.
  • Visual Robustness: The multi-factor decomposition allowed the model to scale effectively to high-dimensional pixel inputs, outperforming methods specifically designed for image-based reward shaping.

Discussion and Insights

One of the primary advantages of TRAIN is its inherent smoothness. Because the graph propagation aggregates information from neighboring nodes, the resulting reward function is less susceptible to the noise and sparsity that often plague manual annotations. This smoothness provides a more stable learning signal for the RL agent.

However, the framework does face limitations. The computational cost of constructing a graph for millions of data points is significant. Currently, the authors address this by "slicing" the dataset into segments of 10,000 to 40,000 pairs. While this heuristic is effective, it relies on the assumption that similar data points are located near each other in the temporal sequence of the recording.

Conclusion

TRAIN represents a significant step forward in making offline RL viable for real-world applications where rewards are scarce. By shifting the problem from global function approximation to transductive graph propagation, the framework achieves high label efficiency and robust performance across diverse control tasks. Future research will focus on more sophisticated data clustering techniques to replace sequential slicing and testing the framework's resilience to noisy, real-world human feedback.

Summary updated

BibTeX
@article{qu2024transductive,
  author = {Bohao Qu and Xiaofeng Cao and Qing Guo and Yi Chang and Ivor W. Tsang and Chengqi Zhang},
  title = {{Transductive Reward Inference on Graph}},
  journal = {IEEE Transactions on Knowledge and Data Engineering},
  year = {2024},
  url = {https://openreview.net/forum?id=0SZnXMWOI9},
}
Figure