TL;DR
This paper argues that for robots to learn quickly and robustly, they should focus on learning forward models (predicting the outcome of an action) rather than inverse models (predicting the action needed for a goal). By using memory-based learning and online numerical inversion, the author demonstrates a system that can master complex tasks—such as a billiards-playing robot—in fewer than 100 trials.
What Problem Does This Paper Address?
In robot control, the standard approach is to learn an inverse model: a mapping that tells the robot, "If you want behavior $B$ in state $S$, perform action $A$." However, inverse models are often "ill-defined" because multiple actions might lead to the same result, or the mapping might be discontinuous. This causes the learner to get "stuck" or fail to improve when the goal is far from its current experience.
Furthermore, traditional neural network approaches often require thousands of trials to achieve adequate performance. In the physical world, robot time is expensive and hardware wear-and-tear is a real constraint. The author argues we should prioritize learning rate (minimizing the number of physical experiments) and robustness over raw computational efficiency, especially as computer hardware becomes increasingly powerful.
Methods
The paper proposes a Memory-Based Forward Model approach to address these challenges:
- Memory-Based Learning: Instead of condensing experiences into global neural network weights, the system explicitly stores every experience as an input-output pair $(State, Action \rightarrow Behavior)$. When a prediction is needed, the system queries this memory using local regression techniques.
- Forward Modeling: The system learns to predict the result of an action rather than the action itself. Because every specific action in a specific state leads to exactly one result, this forward mapping is always well-defined and avoids the "many-to-one" trap of inverse models.
- Online Numerical Inversion: To decide on an action, the robot does not look up a pre-learned answer. Instead, it performs a real-time search (using techniques like hill-climbing or random search) through its forward model to find the action predicted to achieve the desired goal behavior.
- Robustness & Adaptation:
- Robust Regression: The system uses the Median Absolute Deviation (MAD) to identify and ignore "outliers" (noisy or erroneous data).
- Cross-Validation: It automatically tunes internal parameters (like the "width" of local weighting) and implements a "forgetting" mechanism to handle environments that change over time.
Datasets and Tasks
The author evaluates the approach on five distinct simulated and real-world tasks:
- HARD: A 1D task with non-monotonic and discontinuous relationships.
- VARIER: A sinusoidal task where the phase continuously drifts (non-stationary).
- LINEAR: A noisy, high-dimensional (4D) linear relation.
- ARM: A simulated noisy, dynamic two-joint robot arm acting under gravity.
- BILLIARDS: A simulation and a real-world robot task. The robot uses cameras to locate a ball and must swivel a cue stick to sink it into a pocket.
Key Insights and Ideas
- Computation is Cheaper than Experience: The author posits that it is better to spend millions of machine instructions per control cycle if it means the robot learns in 100 physical steps instead of 2,000.
- The Inverse is the Problem: Inverse models are often "many-to-one" (multiple actions for one goal), which confuses standard learners. Forward models are "one-to-one," making them much easier to learn accurately.
- One-Shot Learning: Memory-based methods allow for "aggressive" learning. Unlike incremental gradient descent, the robot can jump directly to the predicted best action based on its entire history of experience.
- Selective Forgetting: By weighting experiences by "age" and using cross-validation to tune that weight, the robot can handle "catastrophic" changes in its environment (like a sensor shifting) without being permanently misled by old data.
Key Results
- Billiards Success: The real-world billiards robot reached a 70% success rate within 100 experiences. For context, the author notes this is "substantially better than the author can achieve."
- Forward vs. Inverse: In almost all tests, the forward model approach significantly outperformed the inverse-only approach. For example, on the "HARD" task, the forward model with cross-validation achieved a success rate of 72% vs. 24% for the basic inverse model.
- Robustness to Noise: The system maintained high performance even in the "BILLIARDS" simulation where 5% of the data were random outliers, thanks to the robust MAD procedure.
Limitations and Future Work
- Computational Scaling: While the paper argues computation is cheap, memory-based queries get slower as the "memory" grows. The author suggests using data structures like kd-trees to maintain speed.
- Search Time: Performing numerical inversion (searching for the best action) can be slow for very high-speed applications. The author suggests a hybrid approach: start the search using a "fast" learned inverse model as an initial guess, then refine it with the forward model.
- Dimensionality: While tested on 4D tasks, the "curse of dimensionality" remains a challenge for memory-based methods in much higher-dimensional state spaces.
Summary updated