Reactive Temporal Logic-based Planning and Control for Interactive Robotic Tasks

Farhad Nawaz, Shaoting Peng, Lars Lindemann, Nadia Figueroa, Nikolai Matni

TL;DR

This paper introduces a modular control architecture that combines high-level discrete task planning with low-level continuous motion control to create robots that are both safe and highly reactive. By integrating temporal logic (for complex decision-making) with Dynamical Systems and Control Barrier Functions (for smooth, safe movement), the researchers demonstrate a Franka robot arm capable of performing interactive tasks—like wiping a whiteboard or a mannequin—while instantly adapting to human interference or environmental changes.

What Problem Does This Paper Address?

Robots operating in human-centric environments face a "dual-speed" challenge. On one hand, they need high-level logic to follow complex instructions (e.g., "If you see a stain, wipe it within 5 seconds; otherwise, continue the normal pattern"). On the other hand, they need low-level reactivity to stay safe and stable when a human bumps into them or moves an object.

Existing solutions often fall into two camps:

  1. Formal Methods: These provide safety guarantees but are often computationally heavy (using mixed-integer programming) and struggle to adapt to unforeseen changes in real-time.
  2. End-to-End Learning: These are flexible but lack the formal safety and task-satisfaction guarantees required for physical human-robot interaction.

The authors aim to bridge this gap by creating a framework that is computationally efficient enough for real-time use (running at 1 kHz) while maintaining formal guarantees that the robot will satisfy its logic-based task specifications.

Methods

The authors propose a modular architecture split into two main layers:

  • Discrete Task Planner (200 Hz): This layer uses Reactive Temporal Logic (RTL). The key innovation here is the distinction between controllable propositions (actions the robot can take, like "wipe left") and uncontrollable propositions (environmental events, like "eraser dropped"). The planner constructs a Büchi automaton (a state machine representing the task logic) and uses a breadth-first search to find the shortest path to complete the task based on current sensor data.
  • Continuous Motion Planner (1 kHz): This layer translates the high-level "behavior" into actual motor commands.
  • It uses Dynamical Systems (DS) to define the shape of the motion (e.g., a periodic wiping pattern).
  • It employs Control Lyapunov Functions (CLFs) to ensure the robot stays on its intended path (stability).
  • It uses Control Barrier Functions (CBFs) to ensure safety and satisfy time-critical constraints from Signal Temporal Logic (STL).
  • Mixing Strategy: To prevent jerky movements when switching tasks, the authors implement a "mixing factor" ($\beta$) that smoothly blends the velocity commands of the old task into the new one.

The entire system is formulated as a Quadratic Program (QP), which can be solved extremely quickly, allowing for the 1 kHz refresh rate.

Datasets

The paper does not use a standard large-scale vision or language dataset. Instead, it relies on:

  • Learning from Demonstration: The nominal motions (the "how-to" of wiping) were learned from 3–4 human demonstrations using Neural Ordinary Differential Equations (Neural ODEs).
  • Experimental Validation: The system was tested on a physical Franka Emika Panda robot arm equipped with an Intel RealSense camera for stain detection and Optitrack markers for tracking human movement.

Key Insights and Ideas

  • Controllable vs. Uncontrollable: By explicitly separating what the robot can control from what the environment does, the system can react to "uncontrollable" events (like a human grabbing the eraser) by instantly re-routing through the Büchi automaton to a "recovery" behavior (like picking the eraser back up).
  • STL as a Constraint, Not a Search: Instead of using heavy optimization to solve Signal Temporal Logic (STL) tasks, the authors translate STL requirements into time-varying CBF constraints. This turns a difficult planning problem into a simple, real-time optimization (QP) problem.
  • Modular Stability: Because the low-level controller uses CLFs and CBFs, the robot remains "compliant." If a human pushes the robot arm, it won't fight back dangerously; it will deviate safely and then return to its task once the perturbation ends.

Key Results

  • Real-Time Performance: The task planner runs at 200 Hz and the motion planner at 1 kHz, which is significantly faster than many traditional formal-methods-based planners.
  • Whiteboard Scenario: The robot successfully switched between left/right wiping, detected a blue stain and prioritized wiping it within a 5-second deadline, and reacted to a "dropped eraser" event by stopping and picking it up.
  • Mannequin Scenario: The robot demonstrated complex logic, such as wiping a mannequin's back only when a human gestured, and interrupting its task to "dip the towel in water" every 30 seconds to keep it wet.
  • Smoothness: The mixing strategy successfully eliminated discontinuities in velocity when switching between different dynamical systems.

Limitations and Future Work

  • Deadlock Handling: The current system assumes that a valid path through the task automaton always exists. If the robot gets stuck (e.g., it needs to pick up an eraser but the eraser is unreachable), the system doesn't currently have a "deadlock recovery" mode.
  • High-Level Supervision: The authors suggest that Large Language Models (LLMs) could be integrated in the future to translate natural language commands into the RTL specifications used by this framework.
  • Complex Manifolds: The current work focuses on Euclidean space. Future work will look at extending these reactive plans to more complex mathematical spaces, such as $SO(3)$ and $SE(3)$, which are necessary for complex tool orientation and manipulation.

Summary updated

Abstract

Robots interacting with humans must be safe, reactive and adapt online to unforeseen environmental and task changes. Achieving these requirements concurrently is a challenge as interactive planners lack formal safety guarantees, while safe motion planners lack flexibility to adapt. To tackle this, we propose a modular control architecture that generates both safe and reactive motion plans for human-robot interaction by integrating temporal logic-based discrete task level plans with continuous Dynamical System (DS)-based motion plans. We formulate a reactive temporal logic formula that enables users to define task specifications through structured language, and propose a planning algorithm at the task level that generates a sequence of desired robot behaviors while being adaptive to environmental changes. At the motion level, we incorporate control Lyapunov functions and control barrier functions to compute stable and safe continuous motion plans for two types of robot behaviors: (i) complex, possibly periodic motions given by autonomous DS and (ii) time-critical tasks specified by Signal Temporal Logic (STL). Our methodology is demonstrated on the Franka robot arm performing wiping tasks on a whiteboard and a mannequin that is compliant to human interactions and adaptive to environmental changes.
BibTeX
@inproceedings{nawaz2024reactive,
  author = {Farhad Nawaz and Shaoting Peng and Lars Lindemann and Nadia Figueroa and Nikolai Matni},
  title = {{Reactive Temporal Logic-based Planning and Control for Interactive Robotic Tasks}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10802335},
}

Loading related content...

Figure