Symbolic State Estimation with Predicates for Contact-Rich Manipulation Tasks

Toki Migimatsu, Wenzhao Lian, Jeannette Bohg, Stefan Schaal

Introduction

In contact-rich manipulation tasks—such as inserting a peg into a hole or connecting a USB cable—robots must navigate a series of complex physical transitions. Success depends on the robot's ability to accurately determine its current state (e.g., "aligned," "in contact," or "inserted") to trigger the appropriate control policy. Traditionally, these transitions are managed by hand-tuned thresholds on force or position sensors. However, these heuristics are notoriously brittle and difficult to generalize across different geometries or environmental conditions.

While deep learning offers a potential solution for state estimation, training end-to-end models typically requires massive datasets and often results in "black box" systems that fail when presented with slightly different tasks. This paper introduces a Bayesian framework that utilizes binary predicates as an intermediate representation, providing a robust, data-efficient, and generalizable approach to symbolic state estimation.

The Predicate-Based Framework

The core innovation of this research is the use of Virtual Predicate Sensors. Instead of attempting to classify a complex symbolic state directly from raw high-dimensional data, the system decomposes the state into atomic binary properties, or predicates (e.g., visual-above(a, b) or motion-force-in-contact(a, b)).

1. Predicate Classifiers

The framework employs modular, specialized classifiers for different modalities. For motion and force data, simple logistic regression models are trained on handcrafted features. For visual data, the authors utilize a ResNet-50 backbone. To ensure data efficiency, this backbone is pre-trained using SimCLR, a self-supervised learning method that allows the model to learn useful visual representations from a small number of labeled examples.

2. Generative Observation Models

To account for sensor noise and classifier uncertainty, the system does not rely on the raw binary output of the classifiers. Instead, it fits Gaussian Mixture Models (GMMs) to the continuous "logits" produced by the classifiers. This allows the framework to model the specific noise characteristics of each virtual sensor, effectively treating the classifier outputs as probabilistic observations.

3. Bayesian Inference and Sensor Fusion

These observations are integrated into a Bayesian filter. The robot maintains a "belief state"—a probability distribution over all possible symbolic states—which is updated recursively as new multimodal data arrives. This structure naturally facilitates sensor fusion; for instance, a state like "contact" can be defined as the logical conjunction of a visual predicate and a force-based predicate, allowing the system to leverage the strengths of different modalities simultaneously.

Experimental Evaluation

The framework was evaluated using the NIST Assembly Task Board I, a standard benchmark for robotic assembly. The study focused on eight distinct insertion tasks, including four electrical connectors (D-sub, RJ45, USB, and a waterproof connector) and four geometric pegs (various rectangular and round shapes).

Notably, the system was trained on a very small dataset of only 120 open-loop policy executions. The observations included 3D positions, velocities, forces, and 500x500 RGB images. This low data volume highlights the efficiency of the predicate-based approach compared to standard deep learning baselines.

Results and Generalization

The proposed method (referred to as Pred) demonstrated significant improvements over both direct state classification and differentiable filter baselines:

  • Offline Accuracy: The predicate-based framework achieved 92% accuracy in state classification, compared to 81% for a direct classifier and 85% for a differentiable filter.
  • Zero-Shot Generalization: When tested on tasks not seen during training, the accuracy of the direct state classifier dropped to 38%. In contrast, the predicate-based method maintained a robust 80% accuracy, demonstrating that predicates like in-contact function as a "universal language" that transfers across different objects.
  • Closed-Loop Performance: In real-world robotic trials, the estimator achieved a 72% success rate in completing insertion tasks. This significantly outperformed manually-tuned expert heuristics (37%), which frequently failed to detect proper alignment during the search phase.

Discussion and Conclusion

The success of this framework lies in its "middle ground" approach. By moving from raw data to predicates and then to symbolic states, the system creates a structured bottleneck that prevents overfitting to task-specific visual or haptic quirks. Ablation studies further revealed that vision and force are complementary: vision excels at detecting if an object is above a target, while force and velocity signals are critical for detecting alignment once physical contact is made.

While the current system requires manual definition of how predicates compose a state, it provides a powerful foundation for robust robotic manipulation. Future work may involve using these symbolic estimates to automatically refine low-level control policies or learning the state-predicate mappings directly from data.

Summary updated

BibTeX
@inproceedings{migimatsu2022symbolic,
  author = {Toki Migimatsu and Wenzhao Lian and Jeannette Bohg and Stefan Schaal},
  title = {{Symbolic State Estimation with Predicates for Contact-Rich Manipulation Tasks}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  doi = {10.1109/icra46639.2022.9811675},
}
Figure