Global Remote Operation of Intelligent Space Robot Assistants

Daniel Leidner, Peter Schmaus, Florian Schmidt, Benedikt Pleintinger, Ralph Bayer, Adrian Simon Bauer, Thomas Krüger, Neal Y. LiiAAAI 2019 · DOI

Abstract

The METERON SUPVIS Justin experiments represent a paradigm shift in space robotics, moving from direct teleoperation to "supervised autonomy." This paper details how astronauts aboard the International Space Station (ISS) successfully commanded a humanoid robot on Earth to perform complex planetary base maintenance. By utilizing object-centric "Action Templates" and an intuitive tablet interface, the system overcomes the challenges of communication latency and high operator workload, demonstrating the feasibility of robots as intelligent coworkers for future lunar and Martian missions.

Introduction

Building and maintaining infrastructure on the Moon or Mars presents extreme hazards for human explorers. While robotic assistants are essential, traditional teleoperation—where a human controls every joint movement in real-time—is mentally and physically exhausting. Furthermore, this "puppeteer" model fails under the significant signal latencies inherent in deep-space communication.

The SUPVIS Justin project addresses these challenges by implementing a supervised autonomy framework. The goal is to elevate the human operator to a supervisory role, deciding what needs to be done, while the robot autonomously determines how to execute the task.

System Architecture and Methods

The technical foundation of the system relies on several integrated components designed to make remote operation intuitive and robust.

Action Templates and PDDL

Instead of raw trajectories, the system uses "Action Templates." These are object-centric representations that combine symbolic logic with geometric data. For instance, a solar panel unit is represented as an object with specific "affordances," such as being cleanable or inspectable. The robot’s reasoning framework utilizes the Planning Domain Definition Language (PDDL) to process these templates, allowing state-of-the-art symbolic planners to generate the necessary sequence of movements to achieve high-level goals.

Intuitive User Interface

Astronauts interact with the robot via a specialized tablet UI. This interface automatically parses Action Templates into a context-aware list of commands. If the robot is positioned near a specific component, the UI highlights that component and presents relevant actions, reducing the need for the operator to memorize complex command sequences.

The SUPVIS Justin tablet interface and the underlying Action Template architecture, combining symbolic and geometric representations.

Experimental Validation

The system was validated through a series of rigorous real-world trials.

ISS Crew Sessions

Between 2017 and 2018, three experimental sessions were conducted involving five NASA and ESA astronauts aboard the ISS. These sessions tested ten distinct protocols, ranging from simple navigation to complex maintenance tasks like replacing computation units and cleaning solar panels.

Global Connectivity: The AAAI 2019 Demo

To demonstrate terrestrial scalability, a real-time benchmark was conducted during the AAAI 2019 conference. A robot located in Munich, Germany, was successfully controlled from Honolulu, Hawaii—a distance of approximately 12,215 km—maintaining stability despite a 250ms round-trip time.

Rollin’ Justin robot in a simulated planetary environment (left) being commanded by an astronaut via a tablet interface (inset).

Key Insights and Results

The experiments yielded several critical insights for the future of space telerobotics:

  • The "Coworker" Paradigm: Offloading low-level reasoning to the robot allows astronauts to focus on scientific objectives rather than joint-level control.
  • Zero-Training Success: The intuitive UI proved highly effective; an astronaut with no prior experience successfully completed the mission's most complex task, validating the system's accessibility.
  • Latency Resilience: Supervised autonomy remained functional even with latencies exceeding 800ms (via the TDRS satellite network), a delay that would render traditional teleoperation impossible.
  • Variable Autonomy: The system allows ground control to adjust the "length" of autonomous action chains, providing a "tight leash" for high-risk tasks and more freedom for routine operations.

Conclusion and Future Work

While the SUPVIS Justin experiments demonstrate a significant leap forward, challenges remain for Mars-scale exploration. Current ISS-to-Earth communication windows are limited to approximately 8 minutes, and Mars-to-Earth delays can reach 20 minutes. Future research must focus on even higher levels of robot autonomy and delay-tolerant commanding. Additionally, the physical hardening of complex humanoid hardware for harsh planetary environments remains a critical engineering hurdle. Nevertheless, the transition to supervised autonomy marks a vital step toward sustainable human-robotic collaboration in deep space.

Summary updated

BibTeX
@inproceedings{leidner2019global,
  author = {Daniel Leidner and Peter Schmaus and Florian Schmidt and Benedikt Pleintinger and Ralph Bayer and Adrian Simon Bauer and Thomas Krüger and Neal Y. Lii},
  title = {{Global Remote Operation of Intelligent Space Robot Assistants}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33019863},
}
Figure