A Differentiable Extended Kalman Filter for Object Tracking Under Sliding Regime

Nicola A. Piga, Ugo Pattacini, Lorenzo NataleFrontiers Robotics AI 2021 · DOI · OpenReview

Introduction

In robotic manipulation, maintaining a stable grip is a fundamental requirement. However, real-world interactions often involve objects sliding through a robot's fingers. Traditional "in-hand" tracking algorithms typically assume a rigid connection between the end-effector and the object, a model that fails immediately upon the onset of slip.

While computer vision systems (e.g., RGB-D cameras) are standard for object tracking, they are frequently hampered by occlusions caused by the robot's own hand during manipulation. Tactile sensors offer a direct alternative, yet they present significant modeling challenges. The complex physics of soft materials, magnetic field interactions, and friction make it nearly impossible to derive precise analytical measurement models. This research introduces a differentiable Extended Kalman Filter (dEKF) designed to track an object's state—position and velocity—during translational sliding using only these complex tactile signals.

Methodology: The Differentiable EKF

The core of this approach is the integration of neural networks (NNs) into the classical Extended Kalman Filter framework. Standard EKFs rely on a motion model and a measurement model. By making these components differentiable, the entire filter can be optimized end-to-end via backpropagation through time (BPTT).

Hybrid Motion and Sensing Models

Rather than relying solely on black-box learning, the authors implemented a hybrid architecture:

  • Hybrid Motion Model: The system utilizes a "constant velocity" physical model for position tracking, while an NN predicts the complex, non-linear changes in velocity that occur during sliding.
  • Derivative-Based Sensing: A key insight was the use of the first derivative of the tactile signal ($\dot{\tau}$) as input. Raw tactile values can be ambiguous (e.g., a static signal could represent either a stationary object or one moving at a constant rate). The derivative focuses the model on the change in sensor state, mapping it directly to velocity.
  • Channel Relevance: Ablation studies revealed that lateral ($xy$) tactile channels are significantly more informative for sliding detection than normal pressure ($z$) channels, which are often dominated by the robot's internal grip control.

Experimental Setup and Dataset

To validate the dEKF, the researchers utilized an iCub humanoid robot equipped with uSkin deformable soft tactile sensors. These sensors consist of a magnet embedded in silicone over a Hall effect sensor, providing high-resolution feedback on surface deformations.

Figure 1: (A) Exploded view of the sensing module. (B) CAD model of the fingertip adapter for the iCub finger. (C) Final assembled fingertip adapters. (D) Experimental setup with the iCub hand and uSkin sensors.

The dataset comprised 100 controlled sliding experiments per object, including a paper box, a plastic water bottle, and a rigid mustard bottle. Ground truth for training was established using an overhead camera and ArUco markers, which were processed through an offline Kalman Smoother to provide clean labels.

Key Results

The dEKF demonstrated high precision and efficiency in real-time scenarios:

  • Tracking Accuracy: The system achieved an average position tracking error of approximately 0.6 cm and a velocity error of 0.05 cm/s.
  • Generalization: Models trained on one object type successfully tracked different objects with only a marginal increase in error (approx. 0.2 cm), suggesting the learned measurement models capture general sliding physics rather than object-specific features.
  • Computational Performance: The pipeline operates at 119 Hz on a standard CPU, enabling its use in high-speed closed-loop control.
  • Closed-Loop Control: In a "stop-on-command" demonstration, the robot successfully allowed an object to slide a precise distance (e.g., 1.5 cm) before increasing grip force to arrest the motion based solely on dEKF estimates.

Conclusion and Future Work

By combining the structural reliability of state estimation with the flexibility of deep learning, the dEKF provides a robust solution for tactile-based object tracking. While currently limited to 1D/2D translational sliding, future work aims to extend this framework to handle 3D rotations and more complex object geometries, eventually integrating it into sophisticated in-hand manipulation controllers.

Summary updated

Abstract

p>Tactile sensing represents a valuable source of information in robotics for perception of the state of objects and their properties. Modern soft tactile sensors allow perceiving orthogonal forces and, in some cases, relative motions along the surface of the object. Detecting and measuring this kind of lateral motion is fundamental to react to possibly uncontrolled slipping and sliding of the object being manipulated. Object slip detection and prediction have been extensively studied in the robotic community leading to solutions with good accuracy and suitable for closed-loop grip stabilization. However, algorithms for object perception, such as in-hand object pose estimation and tracking algorithms, often assume no relative motion between the object and the hand and rarely consider the problem of tracking the pose of the object subjected to slipping and sliding motions. In this work, we propose a differentiable Extended Kalman filter that can be trained to track the position and the velocity of an object under translational sliding regime from tactile observations alone. Experiments with several objects, carried out on the iCub humanoid robot platform, show that the proposed approach allows achieving an average position tracking error in the order of 0.6 cm, and that the provided estimate of the object state can be used to take control decisions using tactile feedback alone. A video of the experiments is available as Supplementary Material.</p>
BibTeX
@article{piga2021differentiable,
  author = {Nicola A. Piga and Ugo Pattacini and Lorenzo Natale},
  title = {{A Differentiable Extended Kalman Filter for Object Tracking Under Sliding Regime}},
  journal = {Frontiers Robotics AI},
  year = {2021},
  doi = {10.3389/frobt.2021.686447},
}
Figure