Abstract
The Multiple View Performer (MVP) is a novel architecture designed for incremental 3D shape completion from sequential 2.5D observations. By integrating linear-attention Transformers (Performers) with a fixed-size associative memory, MVP reconstructs full 3D volumes without explicit point cloud registration. This approach maintains object permanence across occlusions and ensures constant-time updates, making it suitable for real-time robotic manipulation. Evaluations on simulated and real-world datasets demonstrate that MVP achieves high reconstruction accuracy (Jaccard score of 0.90) and improves grasping performance compared to single-view baselines.
Introduction
In robotics, understanding an object's full 3D geometry is critical for planning stable grasps and avoiding collisions. However, standard depth sensors typically provide only a 2.5D "shell" representing the side of the object facing the camera. While multi-view systems can mitigate this, traditional methods often rely on registration—the computationally intensive and noise-sensitive process of mathematically aligning disparate point clouds into a global coordinate frame.
Furthermore, many existing models lack "object permanence." If a robot observes the back of an object and subsequently moves to the front, the model may "forget" the previously seen geometry. The Multiple View Performer (MVP) addresses these challenges by incrementally building a 3D understanding of an object over time, maintaining a memory of occluded parts while remaining efficient enough for real-time deployment.
A key insight of this work is implicit registration: the network learns to fuse information from different viewpoints without explicit knowledge of the spatial relationship between views, effectively learning the underlying logic of 3D shapes.
Methods
The MVP architecture is designed for efficiency and temporal consistency. It processes a sequence of $40^3$ voxel grids and outputs a completed $40^3$ grid representing the full 3D occupancy.
Performers and Associative Memory
Standard Transformers suffer from quadratic computational costs relative to sequence length. MVP utilizes Performers, which employ the FAVOR+ mechanism to linearize attention. This allows the model to process long sequences of views with constant update times.
To manage historical data, MVP incorporates a Compact Associative Memory inspired by modern continuous Hopfield networks. Instead of a growing buffer, the model compresses the history of observations into a fixed-size matrix. This ensures that memory requirements do not scale with the number of observations, a vital feature for continuous robotic operation.
Two-Tower Encoder
The model architecture features two parallel encoding paths:
- Current View Encoder: Processes the immediate observation.
- Prior Views Encoder: Allows the current view to "attend" to the compressed history stored in the associative memory.
This dual-tower approach enables the model to maintain a buffer against occlusion. In "Object Hiding" experiments, the model successfully predicted the full shape of an object even after it was completely obscured, demonstrating robust object permanence.
Experimental Results
The authors evaluated MVP using the YCB Object Dataset (55 household objects) and a larger Grasp Dataset (463 objects). Five "stress test" scenarios were designed, including camera pans, object hiding/revealing, and sliding objects.
Accuracy and Efficiency
MVP was compared against LSTMs and standard Multi-View Transformers (MVT). It consistently performed on par with or better than these baselines in terms of Jaccard similarity (IoU) and F-scores. Notably, in the "Camera Pan" task, MVP achieved a Jaccard score of 0.90. Because of the Performer architecture, the update time for each new frame remained constant, unlike standard Transformers.
Robotic Grasping and Real-World Validation
The utility of the reconstructed shapes was tested in simulated grasping tasks using a BarrettHand. MVP-reconstructed shapes led to significantly lower joint errors (3.83°) compared to single-view reconstructions (4.62°). Finally, the system was validated on a real-world robot using Intel RealSense cameras. Despite being trained entirely in simulation, the model generalized effectively to noisy, real-world depth data.
Conclusion and Future Work
While MVP provides an efficient solution for shape completion, it is currently limited by the $40^3$ voxel resolution, which may be insufficient for fine-grained manipulation of intricate objects. Additionally, the model provides limited utility in "Object Reveal" scenarios until a threshold of information is gathered. Future research may explore continuous representations, such as Signed Distance Fields or Neural Radiance Fields (NeRF), to enhance detail while preserving the efficient temporal memory of the Performer architecture.
Summary updated