Multi-timescale Nexting in a Reinforcement Learning Robot

Joseph Modayil, Adam White, Richard S. Sutton

TL;DR

This paper introduces "multi-timescale nexting" for robots, enabling them to predict future sensory inputs across various time horizons in real-time. By framing predictions as generalized value functions within a reinforcement learning framework, they demonstrate that a robot can learn thousands of such predictions efficiently, achieving accuracy comparable to offline optimal solutions.

What Problem Does This Paper Address?

Humans and animals possess an innate ability to "next," constantly predicting immediate future sensory events. This predictive capability is fundamental to awareness and interaction with the environment. While AI has explored prediction, it often focuses on a limited number of specific, public events or very short-term, myopic predictions. This paper addresses the challenge of enabling robots to perform "nexting" – making a vast number of personalized, short-term predictions across multiple timescales simultaneously, mimicking this fundamental biological capability.

Methods

The core of the approach is a reinforcement learning technique called temporal-difference (TD) learning, specifically TD(λ). Instead of predicting future rewards, the system learns to predict future values of arbitrary sensory signals or state features. Each prediction is treated as a "value function" where the discount rate (γ) is directly tied to the prediction timescale. They employ linear function approximation using a tile-coded feature representation to handle the high-dimensional state space. The learning algorithm updates thousands of prediction weights in real-time using the standard TD(λ) algorithm with eligibility traces. They also extend this to "general nexting" by allowing the discount rate (and thus timescale) to vary based on the robot's current state.

Datasets

The experiments were conducted on a custom-designed mobile robot called the "Critterbot." The robot was equipped with various sensors (motor, light, infrared, acceleration, rotation, magnetic field) providing 53 distinct readings. The data was gathered during a wall-following task in a square pen. No specific pre-existing datasets were used; instead, the robot generated its own experience through interaction with its environment.

Key Insights and Ideas

  • Nexting as Generalized Value Functions: Framing predictions of sensory signals as generalized value functions provides a powerful and unified RL framework for learning anticipatory knowledge.
  • Multi-timescale Prediction is Feasible: The paper demonstrates that it's possible to learn thousands of predictions simultaneously across different timescales (0.1 to 8 seconds) in real-time.
  • Efficiency of TD(λ) with Linear Approximation: The standard TD(λ) algorithm with linear function approximation is computationally efficient enough for real-time robot learning, even with a large number of predictions and features.
  • Single Feature Representation Suffices: A single, well-designed tile-coded feature representation proved effective for learning a diverse set of predictions across various sensors and timescales.
  • General Nexting for Dynamic Timescales: Allowing the discount rate (timescale) to be state-dependent enables more nuanced predictions, such as anticipating power consumption until a specific event or time limit.

Key Results

  • The robot learned 2160 predictions in parallel, updating them at over 10Hz on a laptop.
  • Learning was data-efficient, with significant accuracy achieved within 30 minutes of experience.
  • The learned predictions closely matched the actual future sensory signals, often anticipating changes before they occurred.
  • The accuracy of the learned predictions was comparable to an offline-optimal solution, indicating the effectiveness of the feature representation and learning algorithm.
  • The "general nexting" extension successfully learned to predict motor power consumption based on state-dependent timescales.

Limitations and Future Work

  • Feature Representation: While a single tile-coded representation was effective, the authors acknowledge that performance could be further improved with a better representation. They suggest that determining optimal feature representations remains a challenge.
  • Generalization: The paper focuses on learning predictions from sensor data. Future work could extend these results to more general predictions and, importantly, to robot control.
  • Scalability: While the approach is shown to be scalable to thousands of predictions, extending it to millions or billions of predictions would require further investigation into computational and memory efficiency.
  • Algorithm Tuning: The paper notes that the same learning parameters and representation worked well across many predictions, but formal guidance on parameter selection for new tasks is still lacking.

Summary updated

Abstract

The term “nexting” has been used by psychologists to refer to the propensity of people and many other animals to continually predict what will happen next in an immediate, local, and personal sense. The ability to “next” constitutes a basic kind of awareness and knowledge of one’s environment. In this paper we present results with a robot that learns to next in real time, predicting thousands of features of the world’s state, including all sensory inputs, at timescales from 0.1 to 8 seconds. This was achieved by treating each state feature as a reward-like target and applying temporal-difference methods to learn a corresponding value function with a discount rate corresponding to the timescale. We show that two thousand predictions, each dependent on six thousand state features, can be learned and updated online at better than 10Hz on a laptop computer, using the standard TD(λ) algorithm with linear function approximation. We show that this approach is efficient enough to be practical, with most of the learning complete within 30 minutes. We also show that a single tile-coded feature representation suffices to accurately predict many different signals at a significant range of timescales. Finally, we show that the accuracy of our learned predictions compares favorably with the optimal off-line solution.
BibTeX
@inproceedings{modayil2012multi,
  author = {Joseph Modayil and Adam White and Richard S. Sutton},
  title = {{Multi-timescale Nexting in a Reinforcement Learning Robot}},
  booktitle = {Adaptive Behaviour},
  year = {2012},
  url = {https://openreview.net/forum?id=-0ta39DUM8O},
}
Figure