CAZSL: Zero-Shot Regression for Pushing Models by Generalizing Through Context

Wenyu Zhang, Skyler Seto, Devesh K. Jha

Abstract

Robotic manipulation of novel objects remains a significant challenge due to the complex, non-linear physics involved in interactions like pushing. We present CAZSL (Context-Aware Zero-Shot Learning), a framework that enables robots to generalize physical intuition to entirely new objects without requiring "practice" pushes. By utilizing a Siamese network architecture and a novel context-masking technique, CAZSL adapts its internal parameters based on an object's visual or structural features. Our results on the Omnipush dataset demonstrate that CAZSL achieves zero-shot generalization that outperforms standard meta-learning approaches, reducing prediction error by approximately 10%.

Introduction

Predicting the dynamics of robotic pushing is notoriously difficult. Factors like friction, center of mass, and geometry create complex, non-linear interactions. While deep learning models can be trained for specific objects, they often fail when encountering "out-of-distribution" (OOD) objects—those with different shapes or mass distributions than what was seen during training.

Current solutions typically rely on meta-learning, which requires a few "support samples" (trial pushes) to adapt to a new object. However, for many real-world applications, zero-shot performance—where the robot immediately understands an object's behavior upon sight—is preferred. CAZSL (pronounced "casual") addresses this by learning a continuous relationship between an object's "context" and its physical behavior.

Overview of the CAZSL framework showing how state, action, and context are used to predict the next state via context masking.

Methods

The CAZSL framework is built upon three primary architectural pillars designed to facilitate generalization:

1. Context Masking

Rather than simply concatenating context variables (like shape or weight) with the input, CAZSL uses a separate "context network" to generate a mask. This mask is element-wise multiplied with the activations of the main prediction network. This effectively reconfigures the neural network to be specialized for the specific object being manipulated, allowing the context to fundamentally alter the model's "logic."

2. Siamese Network Structure and Regularization

During training, the model employs a Siamese structure, processing pairs of different objects simultaneously. This allows the model to learn relative differences in behavior based on differences in context. A critical component is Context Regularization, a loss term that ensures objects with similar contexts (e.g., similar geometry) produce similar masks and, consequently, similar predicted behaviors. The authors tested both standard Euclidean ($L_2$) distance and a "Neural Regularization" approach using a small MLP to learn the distance metric.

The Siamese training architecture highlighting context regularization and the prediction objective.

Experimental Evaluation

The researchers evaluated CAZSL using the Omnipush Dataset, a large-scale real-world dataset featuring 250 objects with diverse shapes and mass distributions collected via an industrial robot arm. Two types of context were tested:

  • Indicator Context: A 36-bit binary vector describing the object's properties.
  • Visual Context: $32 \times 32$ top-down images of the objects.

Key Insights

  • Masking vs. Concatenation: Masking was found to be significantly more effective than simple concatenation for physical generalization.
  • The Power of Vision: Using raw images as context consistently outperformed manual feature vectors, as convolutional layers could extract nuanced geometric features that humans might miss.
  • Zero-Shot Efficiency: CAZSL often matched or exceeded the performance of meta-learning baselines like Attentive Neural Processes (ANP), despite the latter requiring extra samples from the new object to adapt.

Results

On the Omnipush dataset, CAZSL reduced the Root Mean Square Error (RMSE) by roughly 9-10% compared to standard fully connected networks. It demonstrated robust generalization to unseen mass distributions and maintained stable prediction errors across various surfaces and shapes (ranging from 4.2mm to 4.8mm), proving its robustness in diverse physical environments.

Conclusion and Future Work

CAZSL represents a significant step toward zero-shot physical reasoning in robotics. However, the current model is restricted to one-step horizon predictions; future work will focus on multi-step trajectory prediction where error compounding is a factor. Additionally, the authors aim to integrate CAZSL into closed-loop controllers to validate its effectiveness in real-time manipulation tasks involving more complex dynamics like tumbling or soft-body manipulation.

Citation: Zhang, W., Seto, S., & Jha, D. K. (2020). CAZSL: Zero-Shot Regression for Pushing Models by Generalizing Through Context. arXiv:2003.11696v2 [cs.LG].

Summary updated

Abstract

Learning accurate models of the physical world is required for a lot of robotic manipulation tasks. However, during manipulation, robots are expected to interact with un-known workpieces so that building predictive models which can generalize over a number of these objects is highly desirable. In this paper, we study the problem of designing deep learning agents which can generalize their models of the physical world by building context-aware learning models. The purpose of these agents is to quickly adapt and/or generalize their notion of physics of interaction in the real world based on certain features about the interacting objects that provide different contexts to the predictive models. With this motivation, we present context-aware zero shot learning (CAZSL, pronounced as casual) models, an approach utilizing a Siamese network architecture, embedding space masking and regularization based on context variables which allows us to learn a model that can generalize to different parameters or features of the interacting objects. We test our proposed learning algorithm on the recently released Omnipush datatset that allows testing of meta-learning capabilities using low-dimensional data. Codes for CAZSL are available at https://www.merl.com/research/license/CAZSL.
BibTeX
@inproceedings{zhang2020cazsl,
  author = {Wenyu Zhang and Skyler Seto and Devesh K. Jha},
  title = {{CAZSL: Zero-Shot Regression for Pushing Models by Generalizing Through Context}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=He13kJliEjT},
}

Loading related content...

Figure