Abstract
This paper introduces a framework for transferring high-level temporal knowledge between Reinforcement Learning (RL) tasks where timing is critical. By inferring Metric Interval Temporal Logic (MITL) formulas from agent trajectories and mapping the underlying "timed automata" between tasks, the authors demonstrate that sampling efficiency in a target task can be improved by up to two orders of magnitude.
Introduction
Reinforcement Learning (RL) is notoriously sample-inefficient, especially in environments with sparse rewards where an agent might wander aimlessly before stumbling upon a goal. While previous research has used propositional or first-order logic to guide RL, these methods often ignore the temporal aspect of tasks—situations where when an action happens is as important as what happens (e.g., "stay in the green zone for at least 5 seconds, then reach the yellow zone within 40 seconds").
The authors tackle three main challenges: formally defining similarity between temporal tasks, inferring implicit task structures from data, and establishing a transfer mechanism for low-level experience (Q-values) when environments or timing requirements differ.
Methodology
The authors propose a two-level transfer learning approach centered on logic as a universal translator.
- Information-Guided Inference: The system collects labeled trajectories from both source and target tasks. It uses a decision-tree-based technique combined with Particle Swarm Optimization (PSO) to infer MITL formulas that best describe the desired behavior.
- Logical Transferability: Tasks are deemed "logically transferable" if the inferred logic for the target task matches the structure of the source task, even if specific time intervals or locations differ.
- Timed Automata & Extended States: MITL formulas are converted into Timed Automata (mathematical models tracking states and "clocks"). The RL agent’s state is then "extended" to include its current location in the automaton and its current clock values, turning complex temporal tasks into reachability problems.
- Q-Function Transfer: By establishing a mapping between the components (clocks, locations, and spatial centroids) of the source and target automata, the agent can initialize its Q-table in the target task using optimal values learned in the source task.
Experimental Evaluation
The framework was evaluated using two simulated Gridworld case studies:
- Case Study 1: A 9x9 grid where source and target tasks have different regions and timing constraints (e.g., staying for 4 units vs. 5 units).
- Case Study 2: A transfer between different environment sizes (5x5 to 7x7) with complex "periodic" rules, such as returning to a base every $N$ seconds.
Key Results
The results show a dramatic "jump-start" in learning efficiency:
- Order of Magnitude Improvements: In Case Study 1, baseline RL took an average of 834,590 episodes to converge. Using the extended state space dropped this to 13,850 episodes, and adding transferred Q-functions further reduced it to just 2,220 episodes.
- Robustness to Scale: In Case Study 2, the agent successfully transferred knowledge from a small grid to a larger one. While the baseline failed to reach optimal performance within 2 million episodes, the transfer-learning agent converged in roughly 6,000 episodes.
Conclusion and Future Work
By augmenting the state with Timed Automata locations, the agent effectively gains a "progress bar" for the task, making sparse rewards much easier to find. While highly effective in Gridworlds, future work is needed to address qualitative similarity metrics, potential inference bias from incomplete data, and scaling to high-dimensional continuous spaces like complex robotics.
Summary updated