GSDet: Gaussian Splatting for Oriented Object Detection

Zeyu Ding, Jiaqi Zhao, Yong Zhou, Wen-Liang Du, Hancheng Zhu, Rui YaoIJCAI 2025 · DOI

Introduction

Oriented Object Detection (OOD) is a critical task in computer vision, particularly for aerial and satellite imagery where objects like ships, vehicles, and aircraft appear at arbitrary rotations. Traditional detectors typically rely on axis-aligned bounding boxes, but OOD requires predicting an orientation angle, significantly expanding the search space.

Current state-of-the-art detectors generally utilize either CNN-based "anchors" or Transformer-based "queries." However, these static candidates often struggle to capture the complex spatial information and varying aspect ratios inherent in rotated objects. GSDet introduces a novel Gaussian-to-box paradigm, formulating OOD as a 3D Gaussian Splatting problem to overcome these limitations.

Comparison of detection paradigms: (a) Anchor-based, (b) Region proposal, (c) Query-based, and (d) GSDet's Random Gaussian approach.

Methodology: The Gaussian-to-Box Paradigm

The core innovation of GSDet lies in representing objects as anisotropic 3D Gaussians within a learned feature space. This approach eliminates the need for manually designed anchors or queries.

3D Feature Space and Representation

Since aerial images are 2D, GSDet constructs a synthetic 3D feature space by treating multi-scale feature maps from a backbone network (e.g., ResNet or Swin Transformer) as "supporting planes" stacked along a virtual Z-axis. Objects are then represented as 3D Gaussians defined by their mean (center), anisotropic covariance (shape and rotation), and a confidence score. These Gaussians are initialized randomly, which surprisingly outperforms grid-based initializations.

Splatting and Transformation

Following the principles of Gaussian Splatting, these 3D Gaussians are projected onto the 2D image plane. The resulting 2D Gaussians are mathematically decomposed into oriented bounding boxes $(cx, cy, w, h, \theta)$. This transformation is naturally suited for OOD because the covariance matrix of a Gaussian aligns elegantly with the geometry of a rotated rectangle.

Comparison between standard 3D Gaussian Splatting (a) and GSDet's OOD approach (b), highlighting the 3D feature space and splatting to image space.

GauAlign and Adaptive Control

To refine these representations, GSDet employs GauAlign, a sampling mechanism that integrates regional features across the Z-axis (feature levels). This allows the model to attend to the most relevant scales for each object, effectively treating "scale" as a spatial dimension. During inference, an adaptive control strategy iteratively refines predictions by keeping high-confidence Gaussians and replacing low-confidence ones with new random candidates, enabling a dynamic "hunting" process.

Experimental Results

GSDet was evaluated on prominent OOD benchmarks, including DOTA-v1.0, DOTA-v1.5, and DIOR-R.

  • DOTA-v1.0: GSDet achieved 75.74% $AP_{50}$ with a ResNet50 backbone, outperforming established query-based and anchor-based methods. With a Swin-T backbone and multi-scale training, performance reached 80.04%.
  • DIOR-R: The model reached 70.05% $AP_{50}$, marking a 0.7% improvement over the baseline.
  • Efficiency: GSDet maintains competitive inference speeds, ranging from 21 to 28 FPS on a ResNet50 backbone, proving faster than many complex two-stage or Transformer-based detectors.

Conclusion and Future Work

GSDet demonstrates that Gaussian Splatting is a powerful and flexible framework for oriented object detection. By treating scale as a spatial dimension and leveraging the mathematical synergy between Gaussians and rotated boxes, it achieves state-of-the-art results without the need for static anchors or queries.

While the current adaptive control strategy is somewhat heuristic, the framework's ability to dynamically refine predictions during inference offers a significant advantage. Future work will focus on refining these update strategies and exploring the framework's application to true 3D data, such as LiDAR point clouds.

Summary updated

BibTeX
@inproceedings{ding2025gsdet,
  author = {Zeyu Ding and Jiaqi Zhao and Yong Zhou and Wen-Liang Du and Hancheng Zhu and Rui Yao},
  title = {{GSDet: Gaussian Splatting for Oriented Object Detection}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/101},
}
Figure