Introduction
In robotic manipulation, maintaining a stable grip is a fundamental requirement. However, real-world interactions often involve objects sliding through a robot's fingers. Traditional "in-hand" tracking algorithms typically assume a rigid connection between the end-effector and the object, a model that fails immediately upon the onset of slip.
While computer vision systems (e.g., RGB-D cameras) are standard for object tracking, they are frequently hampered by occlusions caused by the robot's own hand during manipulation. Tactile sensors offer a direct alternative, yet they present significant modeling challenges. The complex physics of soft materials, magnetic field interactions, and friction make it nearly impossible to derive precise analytical measurement models. This research introduces a differentiable Extended Kalman Filter (dEKF) designed to track an object's state—position and velocity—during translational sliding using only these complex tactile signals.
Methodology: The Differentiable EKF
The core of this approach is the integration of neural networks (NNs) into the classical Extended Kalman Filter framework. Standard EKFs rely on a motion model and a measurement model. By making these components differentiable, the entire filter can be optimized end-to-end via backpropagation through time (BPTT).
Hybrid Motion and Sensing Models
Rather than relying solely on black-box learning, the authors implemented a hybrid architecture:
- Hybrid Motion Model: The system utilizes a "constant velocity" physical model for position tracking, while an NN predicts the complex, non-linear changes in velocity that occur during sliding.
- Derivative-Based Sensing: A key insight was the use of the first derivative of the tactile signal ($\dot{\tau}$) as input. Raw tactile values can be ambiguous (e.g., a static signal could represent either a stationary object or one moving at a constant rate). The derivative focuses the model on the change in sensor state, mapping it directly to velocity.
- Channel Relevance: Ablation studies revealed that lateral ($xy$) tactile channels are significantly more informative for sliding detection than normal pressure ($z$) channels, which are often dominated by the robot's internal grip control.
Experimental Setup and Dataset
To validate the dEKF, the researchers utilized an iCub humanoid robot equipped with uSkin deformable soft tactile sensors. These sensors consist of a magnet embedded in silicone over a Hall effect sensor, providing high-resolution feedback on surface deformations.
The dataset comprised 100 controlled sliding experiments per object, including a paper box, a plastic water bottle, and a rigid mustard bottle. Ground truth for training was established using an overhead camera and ArUco markers, which were processed through an offline Kalman Smoother to provide clean labels.
Key Results
The dEKF demonstrated high precision and efficiency in real-time scenarios:
- Tracking Accuracy: The system achieved an average position tracking error of approximately 0.6 cm and a velocity error of 0.05 cm/s.
- Generalization: Models trained on one object type successfully tracked different objects with only a marginal increase in error (approx. 0.2 cm), suggesting the learned measurement models capture general sliding physics rather than object-specific features.
- Computational Performance: The pipeline operates at 119 Hz on a standard CPU, enabling its use in high-speed closed-loop control.
- Closed-Loop Control: In a "stop-on-command" demonstration, the robot successfully allowed an object to slide a precise distance (e.g., 1.5 cm) before increasing grip force to arrest the motion based solely on dEKF estimates.
Conclusion and Future Work
By combining the structural reliability of state estimation with the flexibility of deep learning, the dEKF provides a robust solution for tactile-based object tracking. While currently limited to 1D/2D translational sliding, future work aims to extend this framework to handle 3D rotations and more complex object geometries, eventually integrating it into sophisticated in-hand manipulation controllers.
Summary updated