Task-Driven Hybrid Model Reduction for Dexterous Manipulation

Wanxin Jin, Michael Posa

Abstract

In contact-rich tasks, like dexterous manipulation, the hybrid nature of making and breaking contact creates challenges for model representation and control. For example, choosing and sequencing contact locations for in-hand manipulation, where there are thousands of potential hybrid modes, is not generally tractable. In this paper, we are inspired by the observation that far fewer modes are actually necessary to accomplish many tasks. Building on our prior work learning hybrid models, represented as linear complementarity systems, we find a reduced-order hybrid model requiring only a limited number of task-relevant modes. This simplified representation, in combination with model predictive control, enables real-time control yet is sufficient for achieving high performance. We demonstrate the proposed method first on synthetic hybrid systems, reducing the mode count by multiple orders of magnitude while achieving task performance loss of less than 5%. We also apply the proposed method to a three-fingered robotic hand manipulating a previously unknown object. With no prior knowledge, we achieve state-of-the-art closed-loop performance in less than five minutes of online learning.

Introduction

Dexterous manipulation—such as a robotic hand rotating a cube—is inherently "contact-rich." Every interaction, whether a fingertip touches, slides, or breaks contact, changes the system's underlying physics. These distinct physical regimes are known as hybrid modes. In typical manipulation scenarios, the number of potential modes is vast, leading to a combinatorial explosion that makes traditional planning and real-time control computationally intractable. While deep learning offers some solutions, it often struggles with the sharp discontinuities characteristic of contact physics.

Methodology: Task-Driven Model Reduction

The core of this work is Task-Driven Hybrid Model Reduction. Rather than modeling every possible physical interaction, the approach identifies a minimal subset of "task-relevant" modes.

Linear Complementarity Systems (LCS)

The robot's dynamics are represented using LCS, a framework that compactly handles the "either/or" constraints of contact (e.g., either a gap exists or a contact force is applied). This allows for a differentiable representation of hybrid dynamics.

Iterative Learning and Control Loop

The method employs an iterative process to refine the reduced-order model:

  1. Train: A violation-based loss function fits a low-dimensional LCS model to data from the rollout buffer.
  2. Control: A Model Predictive Controller (MPC) utilizes this simplified model within a "trust region" to ensure safe and reliable execution.
  3. Rollout: The policy is executed on the robot (or simulator), and the resulting state-action trajectories are added back to the buffer.
The iterative learning loop for task-driven hybrid model reduction, showing the interaction between model learning, MPC control, and rollout data collection.

Experimental Evaluation

The framework was tested on two benchmarks: randomly generated Synthetic Hybrid Systems with up to 30 states and hundreds of modes, and a MuJoCo Simulation of a three-fingered robotic hand performing cube rotation and 3D pose reaching tasks.

Key Results

  • Massive Compression: In synthetic tests, mode counts were reduced from over 100 to just 4 with less than 5% performance loss.
  • Rapid Learning: The robotic hand learned to manipulate unknown objects from scratch in under 5 minutes of online learning (~4,000 samples).
  • Real-Time Control: The reduced models enabled MPC frequencies exceeding 50Hz, facilitating smooth, reactive manipulation.
  • Robustness: The system remained stable even when subjected to external disturbances of up to 5000 rad/s², demonstrating the effectiveness of the closed-loop MPC.

Insights on Mode Sparsity

A critical finding is the sparsity of relevance: while thousands of modes may be theoretically possible, successful task completion often requires only a small fraction (e.g., ~14 active modes). By focusing on "on-policy" data—data relevant to the task at hand—the model avoids the complexity of global accuracy in favor of local task performance.

Visualization of state-space partitioning into task-relevant hybrid modes, illustrating how the complex dynamics are simplified into a few key regimes.

Conclusion and Future Work

This research demonstrates that task-driven reduction can overcome the computational hurdles of hybrid systems in robotics. While the current reliance on Piecewise-Affine (PWA) approximations limits the handling of large nonlinearities, the results on real-time dexterous manipulation are state-of-the-art. Future work will explore Nonlinear Complementarity Systems to handle more complex geometries and larger-scale motions.

Summary updated

Abstract

In contact-rich tasks, like dexterous manipulation, the hybrid nature of making and breaking contact creates challenges for model representation and control. For example, choosing and sequencing contact locations for in-hand manipulation, where there are thousands of potential hybrid modes, is not generally tractable. In this paper, we are inspired by the observation that far fewer modes are actually necessary to accomplish many tasks. Building on our prior work learning hybrid models, represented as linear complementarity systems, we find a reduced-order hybrid model requiring only a limited number of task-relevant modes. This simplified representation, in combination with model predictive control, enables real-time control yet is sufficient for achieving high performance. We demonstrate the proposed method first on synthetic hybrid systems, reducing the mode count by multiple orders of magnitude while achieving task performance loss of less than 5%. We also apply the proposed method to a three-fingered robotic hand manipulating a previously unknown object. With no prior knowledge, we achieve state-of-the-art closed-loop performance in less than five minutes of online learning.
BibTeX
@article{jin2022task,
  author = {Wanxin Jin and Michael Posa},
  title = {{Task-Driven Hybrid Model Reduction for Dexterous Manipulation}},
  journal = {IEEE Trans. Robotics},
  year = {2022},
  doi = {10.1109/tro.2024.3359531},
}
Figure