Abstract
Measurement update rules for Bayes filters often contain hand-crafted heuristics to compute observation probabilities for high-dimensional sensor data, like images. In this work, we propose the novel approach Deep Measurement Update (DMU) as a general update rule for a wide range of systems. DMU has a conditional encoder-decoder neural network structure to process depth images as raw inputs. Even though the network is trained only on synthetic data, the model shows good performance at evaluation time on real-world data. With our proposed training scheme primed data training, we demonstrate how the DMU models can be trained efficiently to be sensitive to condition variables without having to rely on a stochastic information bottleneck. We validate the proposed methods in multiple scenarios of increasing complexity, beginning with the pose estimation of a single object to the joint estimation of the pose and the internal state of an articulated system. Moreover, we provide a benchmark against Articulated Signed Distance Functions (A-SDF) on the RBO dataset as a baseline comparison for articulation state estimation.
Introduction
In robotics, state estimation is fundamental for navigation and manipulation. Bayes filters, such as Kalman or Particle filters, have long served as the standard for tracking a robot's state through prediction and update cycles. However, a significant bottleneck exists in the "update" step: calculating the likelihood of high-dimensional sensor measurements, such as depth images, given a specific state.
Traditionally, this requires a Conditional Probability Density Function (CPDF), $p(y|x)$, representing the probability of measurement $y$ given state $x$. For complex depth maps, standard geometric models often fail to account for occlusions, sensor noise, and cluttered backgrounds. Researchers at ETH Zurich have introduced Deep Measurement Update (DMU) to replace these hand-crafted heuristics with a deep learning framework that processes raw sensor data directly.
The DMU Framework
The core of the DMU approach is a conditional autoencoder (CAE) architecture designed to learn the probability distribution of sensor measurements. Unlike traditional "render-and-compare" methods, DMU uses a neural network to bridge the gap between low-dimensional states and high-dimensional observations.
As shown in the architecture diagram, the system consists of:
- Encoder: Compresses a depth image into a latent code $z$, representing unmodeled environmental factors (e.g., background geometry).
- Conditioning: The system state $x$ (such as a 6D pose) is concatenated with the latent code.
- Decoder: Reconstructs the original measurement from the combined $(z, x)$ pair.
Primed Data Training
A common failure mode in conditional VAEs is the "shortcut" problem, where the encoder hides state information in the latent code, leading the decoder to ignore the condition variable $x$. To circumvent this, the authors developed Primed Data Training. During training, the encoder receives an image $y_1$ with state $x_1$, while the decoder is tasked with reconstructing a different image $y_2$ that shares the same background but has a different state $x_2$. This forces the network to extract only state-independent background information into the latent space.
Experimental Validation and Results
The model was trained exclusively on synthetic data generated via NVIDIA Isaac Sim, yet it demonstrated remarkable zero-shot generalization to real-world datasets, including the RBO dataset for articulated objects.
Key Performance Metrics:
- Efficiency: DMU achieved a 1000x speedup over state-of-the-art Articulated Signed Distance Functions (A-SDF), enabling real-time high-frequency tracking.
- Robustness: The system naturally handles occlusions and multi-modal distributions (symmetries). If an object is partially hidden, the DMU assigns high probability to states consistent with the observed free space.
- Sim-to-Real: Despite the lack of real-world training images, the model successfully tracked objects like laptops and switchboard cabinets in physical environments.
Conclusion
Deep Measurement Updates represent a significant shift in how Bayes filters process complex sensor data. By leveraging conditional autoencoders and the novel primed data training scheme, DMU removes the need for manual heuristics while providing massive gains in computational speed. Future work aims to integrate RGB data and segmentation masks to further enhance performance in highly cluttered or visually complex environments.
Summary updated