TL;DR
This paper introduces iCurb, a novel imitation learning approach for detecting road curbs from aerial images. By framing curb detection as an iterative graph-growing task, iCurb learns to accurately represent road boundaries, offering a more robust and cost-effective alternative to traditional sensor-based methods for autonomous driving.
What Problem Does This Paper Address?
Autonomous driving systems rely heavily on accurately identifying drivable areas on roads. Road curbs serve as crucial visual cues for this purpose. Traditional methods for curb detection typically use vehicle-mounted sensors like cameras or LiDAR. However, camera-based approaches struggle with varying illumination conditions (e.g., nighttime, shadows), while LiDAR can be limited by the sparsity of point clouds, especially for distant curbs. These on-board systems also add to the vehicle's cost and power consumption. The paper addresses this by proposing an off-line detection method using readily available aerial imagery, which offers a wider field of view and can alleviate illumination and sparsity issues. This off-line approach is also valuable for automatically annotating high-definition (HD) maps, a tedious manual process.
Methods
The core of iCurb is an imitation learning framework that treats road curb detection as an iterative graph generation problem. An "agent" navigates an aerial image, starting from an initial vertex candidate, and learns to predict the next vertex along the curb. This process continues until the agent decides to stop, effectively "drawing" a road curb.
Key methodological contributions include:
- Graph Representation: The output is directly a graph (vertices and edges) representing the road curbs, ensuring topological correctness.
- Imitation Learning with DAgger: The agent is trained using a DAgger-based algorithm, which aggregates expert demonstrations and learner's actions to improve robustness and handle exceptions.
- Novel Training Strategy:
- Restricted Exploration: The agent is guided to stay close to ground-truth curb locations to generate accurate training samples.
- Free Exploration: This method allows the agent to explore more freely, helping it learn to handle exceptions and reduce the gap between training and testing distributions.
- Dynamic Labeling: Training labels are generated "on-the-fly" by finding the nearest ground-truth point to the agent's prediction, making the training more robust.
- Agent Network: The agent network consists of two heads: a coordinate head to predict the next vertex's position and a stop head to decide when to terminate curb growing. Historical vertex information is incorporated to provide absolute location context.
- Feature Extraction: A Feature Pyramid Network (FPN) is used as a backbone to extract multi-scale features from the aerial images.
Datasets
The paper utilizes the NYC OpenData NYC-planimetrics dataset [24] for training and evaluation. The authors pre-process this dataset to create their own benchmark.
- The original dataset contains 2,049 high-resolution aerial images.
- These images were split into 25,1000 × 1000 patches.
- After filtering, a pre-processed dataset of 29,000 1000 × 1000 images was created.
- 18,000 images were used for pre-training segmentation networks.
- 10,000 images were used for training iCurb.
- 1,000 images were reserved for testing.
Key Insights and Ideas
- Aerial Imagery for Curb Detection: The paper demonstrates the viability and advantages of using aerial images for off-line road curb detection, overcoming limitations of on-board sensors.
- Imitation Learning for Graph Generation: Framing curb detection as an iterative graph-growing task and using imitation learning is a novel and effective approach for ensuring topological correctness.
- DAgger's Importance: The DAgger algorithm, with its data aggregation strategy, is crucial for training a robust agent that can handle various scenarios and exceptions, leading to better performance than naive behavior cloning or greedy methods.
- Synergy of Exploration Methods: Combining restricted and free exploration strategies in training is key to balancing accuracy and robustness, preventing issues like noisy training data or an inability to handle exceptions.
- Historical Vertex Importance: Including past vertex information in the agent's input is vital for it to understand its absolute position within the image, significantly improving performance.
Key Results
- Accuracy: iCurb achieves state-of-the-art performance in terms of F1-score and connectivity (CC metric) compared to both segmentation-based and graph-based baselines. For instance, iCurb achieves an F1-score of 0.887 and a CC score of 0.866 with a 1.0 pixel threshold.
- Qualitative: Visual results show that iCurb produces more accurate and topologically correct road curb graphs than existing methods, effectively handling complex road structures.
- Efficiency: While training iCurb is computationally intensive (22.38s/image), its inference speed (0.89s/image) is competitive with other graph-based methods.
Limitations and Future Work
The authors acknowledge several limitations:
- Smoothness: The generated road curb graphs can appear less smooth due to the discrete nature of predicted vertices.
- Training Time: The imitation learning and exploration strategies lead to a longer training time.
- Simultaneous Head Failures: iCurb can produce notable errors when both the coordinate and stop heads of the agent make incorrect predictions simultaneously. This is more likely to occur in cases of severe occlusion (e.g., by shadows or plants).
Future work directions include:
- Exploring more powerful backbone networks or multi-modal fusion (e.g., RGB-thermal) to improve robustness against occlusions.
- Developing more efficient training strategies to reduce training time.
- Testing iCurb on new datasets to evaluate its generalization capabilities.
Summary updated