Introduction
In probabilistic planning, systems are often modeled as factored Markov Decision Processes (MDPs)—complex environments where the state is described by numerous interacting variables (e.g., the status of individual computers in a network). Languages like RDDL (Relational Dynamic Influence Diagram Language) allow us to describe these domains symbolically.
The primary challenge is that traditional planners typically solve every new problem instance from scratch. While Deep Reinforcement Learning (RL) has shown promise in learning "reactive policies" for these domains, transferring a learned policy from one instance to another remains difficult. The "meaning" of an action (e.g., "reboot computer #5") changes depending on the underlying topology of the system. Until now, there has been a lack of methods capable of exploiting the symbolic structure of RDDL to transfer knowledge seamlessly across different instances of the same domain.
To address this, researchers have introduced TORPIDO (Transfer of Reactive Policies Independent of Domains), the first domain-independent transfer learning algorithm designed specifically for MDP planning problems expressed in RDDL.
The TORPIDO Architecture
TORPIDO leverages Graph Convolutional Networks (GCNs) and a novel action-decoding architecture to achieve "near zero-shot" transfer across different problem instances. The architecture consists of five main components:
- State Encoder (SE): Utilizes a GCN to treat state variables as nodes in a graph, using RDDL "non-fluents" (fixed properties like network connections) to define edges. This creates a latent embedding of the state that is independent of specific node labels.
- RL Module: An actor-critic agent (based on A3C) that operates entirely within the shared embedding space. Because it processes abstract states rather than raw states, the learned policy is naturally transferable.
- Action Decoder (SAD): Maps the abstract "soft action" embedding outputted by the RL module back to a specific ground action for the current instance.
- Transition Transfer Module (Tr): Learns the domain's transition dynamics in the embedding space. This allows the system to quickly train a new Action Decoder for a target instance using the known RDDL model, even before the RL agent begins learning on the new task.
- Instance Classifier (IC): An adversarial component that attempts to guess which problem instance an embedding originated from. Training against this classifier forces the State Encoder to learn representations that are truly instance-invariant.
Key Insights
- Graph-Based Topology: RDDL provides the connectivity structure of a problem natively. Using a GCN allows the model to treat the domain as a graph, making the policy sensitive to the topology rather than the identity of the nodes.
- Decoupling Policy from Grounding: By training the RL agent in an abstract embedding space and using a separate decoder for specific actions, TORPIDO solves the problem of shifting action meanings between instances.
- Near Zero-Shot Transfer: Because transition dynamics are known via the RDDL file, TORPIDO can pre-train its action decoder for a new instance without ever running a simulator or taking a single step in the environment, providing a massive head start.
- Adversarial Invariance: The instance classifier ensures the model does not overfit to the quirks of the source problems, keeping embeddings useful for unseen target problems.
Experimental Results
TORPIDO was evaluated on three benchmark domains from the International Probabilistic Planning Competition (IPPC) 2014: SysAdmin (managing a network of computers), Game of Life (cellular automata), and Navigation (moving a robot through a hazardous grid). The researchers tested on various instance sizes, ranging from 10 to 50 state variables.
- Superior Learning Curves: In all tested domains, TORPIDO reached high reward levels significantly faster than A3C (learning from scratch) and A2T (a state-of-the-art RL transfer algorithm).
- Zero-Shot Performance: At "Iteration 0" (before any RL training on the target instance), TORPIDO already achieved substantial rewards. In the Navigation domain, its zero-shot score was nearly 70% of the maximum possible, whereas baselines started at 0%.
- Ablation Success: Ablation studies confirmed that the Action Decoder and the GCN were the most critical components. Without the GCN, the model failed to capture domain structure; without the Action Decoder, it could not ground its abstract knowledge.
Limitations and Future Work
Currently, TORPIDO focuses on transferring between problems with an equal number of state variables (though with different connections) and handles single-action selections. Future work aims to scale this approach to transfer across problems of varying sizes (e.g., from a 10-computer network to a 100-computer network), across entirely different domains, and to accommodate combinatorial action spaces.
Summary updated