Abstract
Deep convolutional neural networks (CNNs) based methods have achieved great success in object detection, but when it comes to remote sensing images, new challenges appear considering that current approaches either downsampling or cropping on remote sensing images can hardly provide sufficient structural or semantic information. Aiming to these issues, we propose a multi-branch feature fusion network, in which a global branch and a local branch takes downsampled images and cropped patches as inputs and extracts contextual and structural features respectively. The extracted features are fused and composed into enhanced feature maps. A set of experiments over public building detection dataset for remote sensing images demonstrate the superiority of the proposed method over other state-of-art methods.
Introduction: The "Goldilocks" Problem in Remote Sensing
Detecting buildings in high-resolution satellite imagery presents a unique challenge. Remote sensing (RS) images are often massive, reaching dimensions of 5000x5000 pixels or more. Standard deep learning detectors, such as Faster R-CNN, cannot process these images at full resolution due to GPU memory constraints. This forces researchers into a compromise:
- Downsampling: Shrinking the image makes it manageable but sacrifices structural detail, turning small buildings into blurry, unrecognizable blobs.
- Cropping: Cutting the image into small patches preserves detail but loses global context. Without the surrounding neighborhood, a model may struggle to distinguish a building from a similar-looking truck or shed.
Furthermore, buildings in RS images are often highly clustered and blend into their backgrounds, leading to high false-alarm rates and poor-quality region proposals.
The Multi-Branch Feature Fusion Network
To address these challenges, the authors propose a multi-branch architecture designed to capture both global context and local structural details, effectively achieving the "best of both worlds."
Architecture Overview
The core of the proposed method is a multi-branch system that processes different views of the same scene simultaneously:
- Global Branch: Processes a downsampled version of the entire image using a ResNet-50 backbone to extract high-level semantic and contextual features (the "where").
- Local Branch: Processes high-resolution cropped patches to extract fine-grained structural details (the "what").
Feature Fusion and Enhancement
The "secret sauce" lies in the feature region extraction and fusion process. The model maps the location of high-resolution patches back to the corresponding areas in the global feature map. These features are then fused using $1 \times 1$ convolutions to reduce channel dimensions.
The fused features are subsequently fed into a Feature Pyramid Network (FPN). This allows the model to make predictions across multiple scales, ensuring it can detect both massive industrial complexes and tiny residential sheds. The detection head then performs final classification and bounding box regression.
Experimental Results
The model was evaluated on the XVIEW dataset, a large-scale benchmark containing over 160,000 building instances from WorldView-3 satellites at 0.3m resolution.
Performance Comparison
The proposed method was compared against several state-of-the-art detectors, including Faster R-CNN, Cascade R-CNN, and Libra R-CNN:
- Accuracy: On the XVIEW test set, the proposed method achieved an Average Precision (AP) of 36.54% for patch inference, outperforming Cascade R-CNN (35.14%) and Libra R-CNN (33.24%).
- Global Inference: When processing the entire downsampled image, the fusion method achieved 34.26% AP, a significant improvement over the baseline's 28.31% AP.
- Robustness: In dense urban areas where buildings are highly clustered, the proposed method showed superior robustness and a slower performance decline compared to other models.
Key Insights
- Contextual Knowledge: Incorporating global context (e.g., identifying a residential area vs. a forest) significantly reduces false positives.
- Mitigating Imbalance: By focusing on patches that contain targets, the model effectively manages the foreground-background imbalance common in satellite imagery.
- High-Quality Proposals: The fusion strategy generates better region proposals with higher Intersection-over-Union (IoU) scores compared to standard baselines.
Conclusion
The Multi-Branch Feature Fusion Network successfully bridges the gap between global context and local detail in remote sensing. While the multi-branch approach introduces some computational overhead, the gains in detection accuracy and robustness are substantial. Future work may extend this approach to other objects with weak visual cues, such as small vehicles or maritime vessels in cluttered environments.
Summary updated