Learning Hybrid Object Kinematics for Efficient Hierarchical Planning Under Uncertainty

Ajinkya Jain, Scott NiekumIROS 2019 · DOI · OpenReview

Abstract

Sudden changes in the dynamics of robotic tasks, such as contact with an object or the latching of a door, are often viewed as inconvenient discontinuities that make manipulation difficult. However, when these transitions are well-understood, they can be leveraged to reduce uncertainty or aid manipulation---for example, wiggling a screw to determine if it is fully inserted or not. Current model-free reinforcement learning approaches require large amounts of data to learn to leverage such dynamics, scale poorly as problem complexity grows, and do not transfer well to significantly different problems. By contrast, hierarchical POMDP planning-based methods scale well via plan decomposition, work well on novel problems, and directly consider uncertainty, but often rely on precise hand-specified models and task decompositions. To combine the advantages of these opposing paradigms, we propose a new method, MICAH, which given unsegmented data of an object's motion under applied actions, (1) detects changepoints in the object motion model using action-conditional inference, (2) estimates the individual local motion models with their parameters, and (3) converts them into a hybrid automaton that is compatible with hierarchical POMDP planning. We show that model learning under MICAH is more accurate and robust to noise than prior approaches. Further, we combine MICAH with a hierarchical POMDP planner to demonstrate that the learned models are rich enough to be used for performing manipulation tasks under uncertainty that require the objects to be used in novel ways not encountered during training.

Introduction

Robots operating in human-centric environments frequently encounter objects with "hybrid dynamics"—systems where motion rules change abruptly due to physical constraints, such as a door latching or a drawer reaching its limit. While hierarchical POMDP planning can handle such uncertainty, it typically requires hand-specified models. Conversely, reinforcement learning scales poorly and lacks generalizability. We present MICAH (Model Inference Conditioned on Actions for Hierarchical Planning), a framework to bridge this gap by automatically discovering hybrid automata from noisy demonstrations.

The MICAH Framework

MICAH processes unsegmented motion data through two primary stages to distinguish between intentional stillness and physical constraints.

Act-CHAMP (Action-conditional CHAMP)

Traditional observation-only methods, such as the original CHAMP algorithm, often fail to distinguish between physical constraints and robot inactivity. Act-CHAMP addresses this by conditioning changepoint detection on the robot's actions. If an object remains stationary despite an applied force, the system infers a rigid constraint; if no force was applied, it maintains the current motion hypothesis. This disambiguation is crucial for learning accurate physics from intermittent human demonstrations.

Hybrid Automaton Construction

Once motion modes (e.g., revolute, prismatic, rigid) and their transition points are identified, MICAH formalizes them into a Hybrid Automaton. This mathematical structure defines discrete states (e.g., "Door Open," "Door Latched") and the continuous physics governing each state. These learned models are then integrated into a hierarchical POMDP planner (POMDP-HD), allowing the robot to perform tasks it wasn't explicitly shown during the demonstration phase.

Figure 3: Hybrid automaton representation for a microwave door, showing the transition between revolute and rigid states based on angular constraints.

Experimental Evaluation

The framework was evaluated using kinesthetic demonstrations on three objects: a microwave (revolute/rigid), a drawer (prismatic), and a stapler (revolute/rigid). Tests included "no-action" periods and "high-noise" scenarios (e.g., pushing an object without a firm grasp) to challenge the robustness of the model discovery.

Results and Insights

MICAH demonstrated significant improvements over observation-only baselines across all test cases. In scenarios with intermittent robot activity, Act-CHAMP correctly identified models 70% of the time for the microwave, compared to only 55% for the baseline. In high-noise drawer manipulation, success rates reached 75%, nearly doubling the baseline's 45%.

Figure 6: Learned kinematic models for (a) a microwave and (b) a drawer, illustrating the high-fidelity spatial representations generated by MICAH.

A key insight of this work is that sudden changes in dynamics serve as informational landmarks. For instance, the "click" of a latch provides high-certainty state information even under visual occlusion. Furthermore, because the robot learns a general physics model rather than a specific recipe, it can solve problems creatively. In the stapler experiment, the robot autonomously decided to press the stapler against a table to "lock" it into a rigid, compact shape to navigate a narrow gap—a behavior not present in the training data.

Conclusion

MICAH provides a robust path for robots to learn complex object physics from human interaction. By leveraging action-conditional inference, robots can build the high-fidelity models necessary for sophisticated hierarchical planning in uncertain environments. Future work will explore active exploration strategies to further accelerate model discovery without relying solely on passive human demonstrations.

Summary updated

Abstract

Sudden changes in the dynamics of robotic tasks, such as contact with an object or the latching of a door, are often viewed as inconvenient discontinuities that make manipulation difficult. However, when these transitions are well-understood, they can be leveraged to reduce uncertainty or aid manipulation---for example, wiggling a screw to determine if it is fully inserted or not. Current model-free reinforcement learning approaches require large amounts of data to learn to leverage such dynamics, scale poorly as problem complexity grows, and do not transfer well to significantly different problems. By contrast, hierarchical POMDP planning-based methods scale well via plan decomposition, work well on novel problems, and directly consider uncertainty, but often rely on precise hand-specified models and task decompositions. To combine the advantages of these opposing paradigms, we propose a new method, MICAH, which given unsegmented data of an object's motion under applied actions, (1) detects changepoints in the object motion model using action-conditional inference, (2) estimates the individual local motion models with their parameters, and (3) converts them into a hybrid automaton that is compatible with hierarchical POMDP planning. We show that model learning under MICAH is more accurate and robust to noise than prior approaches. Further, we combine MICAH with a hierarchical POMDP planner to demonstrate that the learned models are rich enough to be used for performing manipulation tasks under uncertainty that require the objects to be used in novel ways not encountered during training.
BibTeX
@inproceedings{jain2019learning,
  author = {Ajinkya Jain and Scott Niekum},
  title = {{Learning Hybrid Object Kinematics for Efficient Hierarchical Planning Under Uncertainty}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  doi = {10.1109/iros45743.2020.9340749},
}
Figure