TL;DR
This paper proposes using semantic segmentation and object detection masks as visual representations for robots navigating to a target object in an unknown environment. This approach allows for effective joint training with both real and simulated data, leading to improved generalization and a higher success rate in navigation tasks compared to previous methods.
What Problem Does This Paper Address?
Navigating complex, unseen environments is a fundamental challenge for intelligent robots. Traditional approaches often rely on building detailed 3D maps and then planning paths, which can be computationally intensive and require accurate localization. More recent data-driven methods learn navigation policies directly from observations, but often struggle with generalization to new environments. This paper specifically investigates what kind of visual representation is most effective for semantic visual navigation, where the robot needs to find a specific object (e.g., a refrigerator) based on its class label. The authors aim to find a representation that is robust, generalizable, and can leverage both real-world and simulated data without the need for complex domain adaptation techniques.
Methods
The core of the proposed method lies in the choice of visual representations. Instead of relying solely on raw RGB or depth images, the authors leverage semantic segmentation masks (identifying object categories per pixel) and object detection masks (bounding boxes for detected objects). These high-level features are extracted using pre-trained, state-of-the-art computer vision models.
These representations, along with the target object's class label and the previous action taken, are fed into a recurrent neural network (RNN) architecture, specifically an LSTM. This network learns a navigation policy by predicting a "cost" for each possible action. The cost is an approximation of the progress made towards the goal, essentially learning to minimize the distance to the target.
A key innovation is the ability to train this system using both real-world data and synthetic data from simulators like SunCG. Because semantic segmentation and object detection masks can be generated from simulated environments (using ground truth labels) and also extracted from real-world data (using off-the-shelf models), the "gap" between real and simulated data is significantly reduced. This allows for joint training without the need for domain adaptation or domain randomization techniques, which are often complex and unstable. The training process uses a form of imitation learning, where the model is supervised by an optimal path planner.
Datasets
- Active Vision Dataset (AVD) [1]: Used for both training and evaluation. It consists of dense scans of real homes. The paper uses specific train/test splits: "similar environments" (different scans of the same home) and "different environments" (entirely different homes).
- SunCG [10]: A large dataset of synthetically generated homes, used to augment the training data.
Key Insights and Ideas
- Semantic representations are key for generalization: High-level semantic information (segmentation and detection masks) proves more effective for generalizing to unseen environments than raw RGB or depth data. This is likely because these representations capture scene layout and object presence, which are more invariant across different environments than raw pixel values.
- Bridging the sim-to-real gap with semantic features: The ability to generate consistent semantic representations from both real and simulated data is a major breakthrough. This bypasses the need for complex domain adaptation, making it easier to leverage vast amounts of synthetic data for training.
- Recurrence matters for navigation: The use of an LSTM component in the navigation policy significantly improves performance. A feed-forward model without a state struggles to correct past mistakes, leading to oscillatory behavior.
- Furniture is important for navigation: When analyzing the importance of different object classes for detection-based navigation, the "Furniture" category (containing large items like couches and tables) was found to be the most critical. These objects are often detectable from afar, providing strong navigational cues.
Key Results
- The proposed model achieves a 54% success rate in navigating to target objects in unexplored environments on the AVD dataset.
- This outperforms a non-learning based baseline (using a map and detector) at 46% and a learning-based baseline at 28%.
- Augmenting training with synthetic data from SunCG improved performance by over 10% for semantic representations (Det and SSeg).
- The use of semantic representations (Det and SSeg) showed better generalization on the "different environments" split compared to RGB-based representations.
Limitations and Future Work
- Depth data's reality gap: While semantic representations benefit from synthetic data, raw depth data's performance drops when trained on both real and synthetic data. This is attributed to the imperfections and noise in real-world depth sensors (like Kinect) compared to perfect synthetic depth. The authors suggest mimicking Kinect's imperfections in synthetic data to mitigate this.
- Implicit history modeling: While the LSTM implicitly models history, the explicit formulation of the cost function doesn't directly incorporate the history of observations.
- Collision detection module: The model assumes the presence of a collision detection module, which is used to inform the policy about recovery actions.
- Future work: The paper suggests further investigation into combining different modalities and exploring more sophisticated ways to leverage synthetic data. The authors also highlight the potential for this approach in other embodied AI tasks.
Summary updated