Robot Learning: Exploration and Continuous Domains

David A. CohnNeurIPS 1993

Introduction

Traditional machine learning algorithms often thrive in discrete, simulated environments, but real-world robotics operates in messy, continuous domains where exploration is expensive and potentially dangerous. This paper summarizes a pivotal workshop addressing the fundamental friction between theoretical abstractions and the physical reality of robot learning. The core takeaway is a necessary shift away from "monolithic" learning models in favor of hybrid strategies that prioritize safety, standardized domain assumptions, and the ability to transfer knowledge across multiple tasks.

The authors identify three primary gaps hindering the field:

  1. The "Assumption" Problem: Researchers frequently rely on idiosyncratic assumptions about their robot's environment, making it nearly impossible to compare different techniques fairly.
  2. Catastrophic Failure: A robot that "learns" by crashing into a wall ten times is a failure in a laboratory setting. There is a critical need for learning strategies that can push performance limits without breaking the hardware.
  3. The One-Task Trap: Most robot learning historically focuses on mastering a single, isolated task, which is highly inefficient for developing general-purpose autonomous agents.

Methodological Innovations

To bridge the gap between simulation and reality, workshop participants proposed several novel methodologies designed specifically for continuous domains:

  • Locally Weighted Regression: This technique enables the "spatial mixing" of controllers. Rather than relying on one giant, complex model, a robot can use different local models tailored to different parts of its state space.
  • Advantage Updating: A reinforcement learning (RL) algorithm designed explicitly for continuous-time and noisy domains, addressing the frequent failures of standard RL in these settings.
  • Coaching Signals: A method designed to accelerate learning by mixing naive feedback controllers with "coached" signals.
  • Truth Maintenance Systems: An AI-inspired approach that allows robots to dynamically switch between different sets of domain assumptions based on their empirical validity during a live task.
  • Information Gain as Reinforcement: A strategy to solve the classic exploration-exploitation tradeoff by using the acquisition of new information as a direct reward signal.

(Note: As a theoretical workshop summary, specific datasets were not the primary focus, though platforms like the Siemens RatBot were highlighted for landmark-based navigation experiments.)

Key Insights and Theoretical Advances

The workshop yielded several conceptual breakthroughs and points of consensus regarding how robots should learn and operate safely:

The "Goldilocks" of Assumptions: Andrew Moore argued that neither "fewer" nor "more" assumptions are inherently better. Instead, the field desperately needs standardized assumption sets to make experimental results comparable. He proposed algorithms capable of detecting when an assumption set is failing and autonomously switching to a more appropriate one.

Safety via Suboptimality: Satinder Singh introduced a clever workaround for the inherent dangers of physical exploration: using reinforcement learning to mix two "safe" but suboptimal controllers. This allows the robot to incrementally improve its performance without ever entering a high-risk state.

Supplanting Instinct: José del R. Millán presented a hierarchical learning structure where a neural controller gradually takes over from a robot's basic "reactive instincts" (such as innate obstacle avoidance) as it becomes more proficient. This ensures a baseline safety net is always active during the learning process.

Rejection of Monolithic Approaches: A strong consensus emerged against using single, massive controllers. The future of the field lies in "mixing" strategies—whether spatially, temporally, or hierarchically.

Limitations and the Path Forward

While the workshop outlined a clear trajectory for robot learning, several significant hurdles remain.

First, participants universally agreed that "one-task learning" is no longer a suitable goal and that future robots must share concepts, biases, and models across multiple tasks. However, the field still lacks a concrete, universal definition of what "task transfer" actually entails.

Second, despite new algorithms like advantage updating, exploration in continuous space—especially in the presence of high noise—remains an intractable challenge. The consensus suggested that the robotics community must look toward the "dual control" literature in traditional control theory to better balance the gathering of data (exploration) with the execution of tasks (exploitation).

Finally, the lack of commonality in domain assumptions remains a severe barrier to scientific progress. Overcoming this will require a cultural shift among researchers to adopt shared frameworks and standardized benchmarks.

Summary updated

BibTeX
@inproceedings{cohn1993robot,
  author = {David A. Cohn},
  title = {{Robot Learning: Exploration and Continuous Domains}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1993},
}
Figure