Abstract
The Offline Skill Graph (OSG) framework addresses the challenges of deploying reinforcement learning in real-world robotics by enabling complex, long-horizon task planning using libraries of skills learned entirely from static datasets. By integrating offline reinforcement learning (ORL) with a state-to-skill classifier and a structured skill graph, OSG allows robots to navigate from arbitrary initial states to goals without requiring dangerous or time-consuming real-world exploration. Experimental results in a simulated kitchen environment demonstrate high skill robustness (95-100% success) and effective long-horizon task execution (78% success), highlighting the potential for explainable and safe robotic learning.
Introduction
Reinforcement Learning (RL) has achieved significant success in simulations, but its transition to real-world robotics is hindered by three major hurdles: dangerous exploration, a lack of generalization across tasks, and the "black box" nature of deep policies which complicates safety-critical deployment.
While Offline RL (ORL) addresses exploration risks by learning from pre-collected data, it typically focuses on single, specific tasks. Conversely, Deep Skill Graphs (DSG) facilitate planning over multiple tasks but usually require extensive online exploration to construct the graph. The Offline Skill Graph (OSG) bridges this gap by creating a purely data-driven framework for both learning primitive skills and planning the transitions between them.
Methodology
The OSG framework consists of three primary modules designed to transform static data into executable plans:
- ORL Skills Module: Primitive "skills" (e.g., reaching, grasping, lifting) are learned from static datasets using state-of-the-art offline algorithms like TD3+BC. These skills are framed as "options"—mathematical structures consisting of a policy, an initiation set, and a termination condition.
- Skills Graph Module (SGM): This module chains learned options into a directed graph. An edge is formed if one skill's termination state matches another's initiation state, allowing a high-level planner to find sequences of skills to reach distant goals.
- State Classifier Module: To handle the "entry point problem," this neural network classifies the current world state to determine which skill is most likely to lead a robot onto the graph from an arbitrary starting position.
Experimental Setup and Datasets
The framework was evaluated in a PyBullet simulation featuring a Franka Emika 7-DOF robotic arm in a kitchen environment. Rather than using expert demonstrations, the researchers utilized "scripted noisy policies" with mediocre success rates (40-50%) to generate 10,000 simulation episodes. This approach tests the framework's ability to outperform its training data. The data covered 12 distinct skill sets, including basic movement, cabinet door interaction, and cube manipulation.
Key Results and Discussion
OSG demonstrated high reliability across several metrics:
- Skill Robustness: Individual skills achieved success rates of 95% to 100% in noise-free environments. As shown in the performance data, many skills maintained near-perfect reliability even when subjected to spatial noise.
- Classifier Accuracy: The state-to-skill classifier achieved a corrected accuracy of 92.7%, proving it can effectively direct the robot toward the skill graph from random starting points.
- Long-Horizon Success: In complex combined tasks—such as opening a door, picking up a cube, and placing it inside a cabinet—the framework achieved a 78% success rate without any online fine-tuning.
A key insight is the framework's explainability; because the robot follows a sequence of discrete skills, failure points are significantly easier to identify and debug compared to end-to-end neural networks.
Conclusion and Future Work
OSG provides a robust, data-driven path for robotic planning. However, the system currently faces sensitivity to Inverse Kinematics (IK) configurations and requires individual policies for each skill. Future research will focus on goal-conditioned policies to reduce model overhead and validating the framework on physical hardware to address sim-to-real challenges such as sensor noise and complex physics.
Summary updated