Integrating Human Expertise in Continuous Spaces: A Novel Interactive Bayesian Optimization Framework with Preference Expected Improvement

Nikolaus Feith, Elmar Rueckert

Abstract

Traditional machine learning approaches, particularly Reinforcement Learning (RL) and Bayesian Optimization (BO), often struggle with the "curse of dimensionality" and high sample complexity when applied to real-world robotics. While humans possess significant physical intuition, they often lack the precision required to tune high-dimensional motor parameters. This paper presents a novel Interactive Bayesian Optimization (IBO) framework that integrates human expertise directly into the optimization loop. By utilizing a new acquisition function called Preference Expected Improvement (PEI), the framework allows users to shape policies in continuous spaces through manual adjustments and dimensional preferences. Experimental results across simulated benchmarks and real-world robotic tasks demonstrate that IBO significantly enhances sample efficiency and optimization performance compared to standard autonomous baselines.

Introduction

In the pursuit of efficient robotic learning, the "human-in-the-loop" paradigm offers a promising bridge between human intuition and algorithmic precision. However, most existing Interactive Machine Learning (IML) systems are constrained to discrete feedback or binary choices, which are often insufficient for the complex, continuous action spaces inherent in robotics.

Nikolaus Feith and Elmar Rueckert propose a solution to this limitation with their Interactive Bayesian Optimization (IBO) framework. The core objective is to allow human experts to guide the optimization process in high-dimensional spaces, effectively pruning the search space and steering the algorithm toward promising regions that a purely stochastic search might take thousands of iterations to discover.

The IBO Framework and Preference Expected Improvement

The proposed framework is built upon Bayesian Optimization using Gaussian Processes (GP) as surrogate models. The primary innovation lies in the Preference Expected Improvement (PEI) acquisition function, which mathematically formalizes the integration of human feedback.

The Interaction Loop

The IBO system operates through a structured interaction cycle:

  1. Interaction Trigger: The system determines when to solicit human intervention based on specific metrics, such as regular intervals, random triggers, or when the algorithm's improvement rate plateaus.
  2. Human Intervention: When prompted, the human expert interacts with a Graphical User Interface (GUI) displaying the current best-performing policy. The expert provides two types of input:
  3. Shaping: The user manually adjusts policy parameters (e.g., modifying the height of a robotic arm's trajectory).
  4. Preference: The user identifies specific dimensions or parameters that are already "optimized," signaling the algorithm to focus its exploration efforts elsewhere.
  5. PEI Integration: Unlike standard BO, which typically uses a uniform distribution for sampling, PEI utilizes a proposal distribution centered on the human's manual adjustments. This effectively blends the human's "educated guess" with the algorithm's statistical uncertainty, shrinking the effective search space to a manageable region.

To ensure practical utility, the system was implemented using ROS2, providing a modular architecture suitable for real-time robotic applications.

Experimental Evaluation

The researchers evaluated the IBO framework across three distinct environments, each designed to test different aspects of the human-algorithm collaboration.

1. High-Dimensional Cartpole (OpenAI Gym)

To test the framework's resilience to the curse of dimensionality, the authors scaled the classic Cartpole balancing task to 15 dimensions. In this environment, standard BO performance typically flatlines early. However, IBO variants—particularly those utilizing the "Mixture" approach of shaping and preference—continued to improve, demonstrating that human feedback can effectively mitigate the difficulties of high-dimensional search.

2. Reacher (DeepMind Control Suite)

The Reacher task required controlling joint accelerations to move a simulated arm to a target. This environment highlighted the "Intuition Gap": because joint acceleration is not an intuitive space for human movement planning, users found it more difficult to provide helpful feedback compared to Cartesian tasks. Despite this, IBO still outperformed standard BO, albeit with higher variance than in more intuitive tasks.

3. Real-World Robotics (Franka Emika Panda)

The framework was deployed on a physical Franka Emika Panda robot tasked with learning a 2D trajectory while avoiding a designated exclusion zone. The use of a Cartesian coordinate interface allowed the human expert to provide highly effective shaping feedback, enabling the robot to reach high-reward policies with significantly fewer physical trials than autonomous methods.

Key Insights and Results

  • The Superiority of Hybrid Feedback: The "Mixture" strategy—combining manual parameter shaping with dimensional locking—consistently yielded the highest mean rewards and the most stable learning curves.
  • Sample Efficiency: IBO reached optimal or near-optimal policies in a fraction of the episodes required by non-interactive baselines. This is a critical advantage for physical robotics, where reducing trial counts minimizes mechanical wear and operational costs.
  • Task Representation Matters: The success of human-in-the-loop systems is heavily dependent on how the task is visualized. Interfaces that align with human physical intuition (like Cartesian coordinates) result in much more effective intervention than abstract parameter spaces (like joint accelerations).

Conclusion and Future Work

The IBO framework represents a significant step forward in making Bayesian Optimization viable for complex, real-world robotic tasks. By allowing humans to "prune" the search space through PEI, the system overcomes traditional dimensionality limits.

Future research will focus on improving the visualization of high-dimensional models to help humans better understand the robot's internal state. Additionally, the authors aim to refine the "Interaction Metric" to optimize the timing of human interventions, ensuring that the expert's effort is requested only when it will provide the maximum benefit to the optimization process.

Summary updated

BibTeX
@inproceedings{feith2024integrating,
  author = {Nikolaus Feith and Elmar Rueckert},
  title = {{Integrating Human Expertise in Continuous Spaces: A Novel Interactive Bayesian Optimization Framework with Preference Expected Improvement}},
  booktitle = {UR},
  year = {2024},
  url = {https://openreview.net/forum?id=5PR8Sx7jsc},
}
Figure