Abstract
While modern computer vision systems excel at mapping the geometry and appearance of environments, they remain largely oblivious to the underlying physical properties of objects. This paper presents an autonomous robot experimenter capable of learning dense, high-fidelity maps of physical properties—such as rigidity, material composition, and friction—through active exploration. By combining real-time 3D neural fields with entropy-guided interaction, the system transforms a limited number of physical probes into a comprehensive scene representation from scratch.
Introduction
In robotics, vision is often "blind" to the physical realities of the world. A robot may perceive a pile of laundry and a pile of rocks as geometrically similar, yet the strategies required to interact with them are diametrically opposed. Traditional approaches to physical scene understanding rely on massive pre-trained datasets to infer properties from visual cues. However, these models often fail in novel or unpredictable environments where visual appearance does not correlate perfectly with physical behavior.
This research introduces a framework for autonomous physical scene understanding. Instead of relying on prior guesses, the robot acts as a scientist: it explores an unknown environment, conducts physical experiments (poking, pushing, and scanning), and builds a rich internal representation of the scene's physical landscape in real-time.
Methodology
1. Neural Field Representation
The system utilizes a foundation of Neural Fields, specifically building upon architectures like iMAP and iLabel. The scene is represented by a single Multi-Layer Perceptron (MLP) that maps 3D coordinates to color, volume density, and a specialized "semantic head." This semantic head is responsible for storing and interpolating the physical properties discovered during exploration.
2. Entropy-Guided Active Exploration
Efficiency is critical in physical robotics, as every interaction takes time and energy. Rather than sampling the environment randomly, the robot calculates the softmax entropy across the neural field to identify regions of maximum uncertainty. By targeting these high-entropy areas, the robot prioritizes experiments that provide the most information about the scene's physical makeup.
3. Multimodal Physical Interaction
The robot employs three primary modes of interaction to gather ground-truth data:
- Poking: Top-down interactions equipped with force sensing allow the robot to classify objects as rigid or non-rigid.
- Spectroscopy: A single-pixel multiband spectrometer identifies material types (e.g., distinguishing between wool, cotton, and synthetics) by analyzing light reflectance.
- Pushing: Lateral movements measure "stiction" (static friction) and mass distribution, allowing the robot to infer how much force is required to move an object.
4. Real-time Optimization
As the robot collects sensor data, it backpropagates this information into the MLP. A key advantage of neural fields is their inherent smoothness; a single interaction in one location allows the model to propagate that physical label across the entire geometrically coherent object, leading to rapid convergence of the scene map.
Experimental Results
Segmentation and Classification
The system was evaluated on complex tabletop scenes, including mixed-material laundry piles and containers of varying hardness. In these challenging environments, the autonomous approach achieved a Mean Intersection over Union (mIoU) of 0.89. This significantly outperformed state-of-the-art class-agnostic baselines like Mask R-CNN and Unseen Clustering Network (UCN) + RICE (0.56), which struggled to find boundaries in the absence of clear depth or color contrasts.
Continuous Property Mapping
Beyond discrete classification, the system successfully mapped continuous physical variables. For instance, when interacting with a power drill, the robot generated a "force map" that correctly identified the non-uniform mass distribution of the tool—recognizing that the battery base required more force to move than the handle. These results closely aligned with analytical physics models.
Efficiency and Curiosity
The entropy-guided strategy proved highly effective. In one experiment, the robot successfully categorized a complex pile of laundry into three distinct material types using only 46 targeted interactions. The curiosity-driven approach reached high accuracy levels significantly faster than random sampling methods.
Discussion and Key Insights
- Model Coherence: The research demonstrates that the MLP’s joint learning of shape and semantics is a powerful tool. Geometric boundaries act as natural containers for physical labels, allowing the robot to understand an entire object's properties after touching only a small portion of it.
- Temporal Memory: An interesting emergent property is the network's "temporal memory." Even when an object's geometry is altered (e.g., by being pushed), the network weights retain information about the object's material, even if the visual history is cleared to prevent tracking errors.
Limitations and Future Work
While robust, the current system assumes that objects being pushed are rigid and that rigid objects are not obscured by flexible ones. Additionally, the robot currently generates interaction queries from a fixed initial viewpoint, which may limit its ability to map the "back side" of objects. Future research will focus on expanding the robot's experimental toolbox to include grippers for graspability testing and thermal sensors for measuring heat conductivity, as well as improving error recovery through confidence-weighted measurements.
Summary updated