Fusing RGBD Tracking and Segmentation Tree Sampling for Multi-Hypothesis Volumetric Segmentation

Andrew Price, Kun Huang, Dmitry Berenson

Introduction

Robotic interaction in cluttered environments is often hindered by the inherent ambiguity of 3D instance segmentation. When objects are in close proximity or partially occluded, a single camera view may not provide sufficient information to distinguish between individual items. Most existing algorithms produce a single "best guess," which, if incorrect, can lead to manipulation failure. This research proposes Multihypothesis Segmentation Tracking (MST), a system that explicitly represents uncertainty by maintaining multiple plausible hypotheses and updating its beliefs as new information—such as motion—clarifies object boundaries.

Methodology

The MST framework operates as a recursive pipeline designed to evolve a population of scene hypotheses over time. The process integrates four primary stages:

  1. Segmentation Sampling: The core of MST's ability to handle uncertainty lies in its sampling strategy. Rather than committing to a single maximum-likelihood segmentation, the system employs a Metropolis-Hastings (MCMC) sampler to traverse a hierarchical "segmentation tree" (specifically an Ultrametric Contour Map). By using the SceneCut quality metric, the sampler can generate a diverse population of plausible 2D segmentations. This approach ensures that even if the most likely guess is wrong, the correct configuration is likely present within the set of hypotheses.
  2. Volumetric Shape Completion: Each 2D segment is processed by a deep neural network to estimate its 3D volume, "hallucinating" occluded parts of the object to create a full volumetric representation.
  3. Tracking and Fusion: As the robot interacts with the scene, SiamMask provides visual tracking, while SIFT and ICP handle geometric tracking. These tools move the 3D hypotheses in sync with observed motion, maintaining temporal consistency across frames.
  4. Conflict Resolution: New sensor data is reconciled with existing hypotheses using a conflict graph approach. This allows the system to merge or split object representations based on their consistency with ongoing observations, effectively "evolving" the hypothesis pool.
The MST recursive pipeline: combining segmentation sampling, shape completion, and tracking to evolve scene hypotheses.

Experimental Evaluation

The system was evaluated in both simulated (Gazebo) and real-world environments. Simulations involved scenarios with increasing clutter where a robot performed manipulation actions such as pushes and pulls to create dynamic scenes. Real-world tests utilized common household objects from the YCB dataset, introducing challenges like sensor noise, varying lighting conditions, and complex textures.

Results and Discussion

MST consistently outperformed single-frame and tracking-only baselines in terms of match quality, measured by symmetric weighted coverage. A key strength of the framework is its ability to use motion as a clarifier; rigid-body motion during manipulation makes object boundaries more apparent, allowing MST to prune incorrect hypotheses.

Extracted figure from page 1 of Fusing RGBD Tracking and Segmentation Tree Sampling for Multi-Hypothesis Volumetric Segmentation

In real-world experiments, the system demonstrated the ability to recover from initial errors, such as oversegmentation, by observing objects move as single units. Furthermore, by retaining 3D information from previous frames, MST maintained robust object representations even under subsequent occlusion, "remembering" object shapes that were later hidden behind other items.

Conclusion

MST represents a significant step toward robust robotic perception in complex scenes. By fusing probabilistic sampling with temporal tracking, it provides a mathematically grounded framework for managing uncertainty in 3D volumetric segmentation. While the current implementation faces computational bottlenecks—taking roughly one hour to process short sequences—future work focusing on parallelization and the incorporation of physics-based priors promises to make this approach viable for real-time robotic applications.

Summary updated

BibTeX
@inproceedings{price2021fusing,
  author = {Andrew Price and Kun Huang and Dmitry Berenson},
  title = {{Fusing RGBD Tracking and Segmentation Tree Sampling for Multi-Hypothesis Volumetric Segmentation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  doi = {10.1109/icra48506.2021.9561136},
}
Figure