Many object detectors improve their boxes by running them through a cascade: a first prediction is fed to a second and third "refinement stage," each correcting the previous one. Every box pays for every stage, but most of the accuracy gain shows up in the first one or two corrections, and much of it is concentrated in a minority of objects. This is sharpest in rotated-box detection on aerial imagery. There, a box is five numbers: center, width, height and angle. For a long thin ship, a few degrees of angle error destroys the overlap with the true box. For a square storage tank, angle barely matters. We propose to predict, for each box after a cheap first stage, how much overlap the next stage would actually add. Only boxes with a large predicted gain are sent onward. The predictor is fed the box's geometry and its predicted joint uncertainty over the five numbers, not one quality score.
The two-sentence version is this. Treating the five box parameters as independent targets hides the fact that whether a box can be improved depends on how its errors combine with its shape. We use that coupled geometry to decide, per box, whether to spend another refinement stage, aiming for the high-precision accuracy of a full cascade at a fraction of its refinement cost.
Where this sits
Three lines of prior work frame the problem, and each plays a different role.
The first line supplies the coupled box losses we reuse. Losses that score a rotated box as a whole replace separate per-coordinate penalties. KLD models boxes as 2D Gaussians and shows that the angle gradient is automatically scaled by aspect ratio. It lifts HRSC2016 $AP_{75}$ from 48.42% to 72.39% on RetinaNet. KFIoU builds an overlap surrogate that is differentiable everywhere from the product of two Gaussians, reaching 77.35% mAP on DOTA-v1.0. It also documents a limit: near-square boxes lose their heading. ARS-DETR weights angle errors by aspect ratio and argues that $AP_{50}$ hides angle errors that $AP_{75}$ exposes. It names inference speed as its main limitation.
The second line motivates the routing idea. Cascades help high-precision localization but spend compute uniformly. Cascade R-CNN more than doubles COCO $AP_{90}$, from 7.1 to 15.8, and reports negligible gains beyond three stages. FSCascade shows that the third stage is actually worse than the second on loose boxes (−1.1 $AP_{50}$). That suggests forcing every box through every stage can hurt, not just waste time. R3Det adds a feature re-sampling refinement stage worth 2.8% mAP on DOTA, applied to all candidates. IoU-Net refines by gradient ascent on a learned overlap predictor with a fixed number of steps, raising per-image time from 0.255s to 0.305s.
The third line motivates the gain-based view from outside vision. iRe2f shows in parsing that errors are sparse, so re-processing everything at each refinement pass is wasteful. Its cheaper refiner lifts throughput from 98 to 152.5 sentences/sec. Meta-Reflection compresses iterative reflection into a single pass for language models.
What none of these do is decide, per instance, whether refinement will help. This also clarifies what we are not claiming. Putting a coupled loss inside an oriented cascade is not new in substance: the toolkit MMRotate already pairs two-stage rotated detectors with KLD and reports 80.90% mAP on DOTA-v1.0. That combination is our baseline, not our contribution.
Core Claim
In an oriented-box cascade trained with a coupled loss, the improvement each later stage delivers is concentrated and predictable from first-stage geometry. A small head can predict that per-box gain from three inputs: the predicted Gaussian (center $\mu$, shape covariance $\Sigma$), a predicted error covariance over the five parameters, and pooled features. Routing on this prediction should recover nearly all of a fixed three-stage cascade's $AP_{75}$ while running later stages on a minority of boxes.
We also claim that the learning target matters. A detector's estimate of how good a box currently is differs from how much better the next stage will make it. Small or occluded boxes can be poor yet unfixable, because the next stage sees the same weak evidence. Moderately good elongated boxes can sit just below the 0.75 overlap threshold and be fixed by a two-degree rotation.
Research Program
We build an oriented cascade in MMRotate using rotated RoI feature extraction and three heads trained with KFIoU. Each head outputs, besides the box, a full 5×5 error covariance trained by Gaussian likelihood on its own residuals. This extends the idea in Gaussian YOLOv3 of modelling uncertainty per coordinate to correlated errors.
After stage $k$, a gain head $g_k$ predicts
$$\hat{\Delta}_k = g_k(\mu, \Sigma, C, f) \approx \mathrm{IoU}(b_{k+1}, b^*) - \mathrm{IoU}(b_k, b^*),$$
where $b_k$ is the stage-$k$ box, $b^*$ is the ground truth, $C$ is the predicted error covariance, and $f$ is the pooled feature. The training target is the realized gain of the frozen cascade on held-out training images, so the gain head does not memorize its own training boxes. At test time, boxes with $\hat{\Delta}_k$ below a threshold $\tau$ exit. $\tau$ is set on validation data to hit a target fraction of routed boxes, which traces an accuracy–cost curve.
Routing is a gather over boxes, so it batches cleanly on a GPU. That avoids the variable-length episodes that hamper the step-by-step box agent in BAR.
We also run an analysis track. Before any routing, we measure on DOTA-v1.0 and HRSC2016 how per-stage gain is distributed across aspect ratio, object size, first-stage overlap and predicted error covariance. We compare these distributions for cascades trained with Smooth $L_1$ and with KFIoU. This answers whether coupled losses shrink and concentrate the gain from later stages, which is the premise everything else rests on.
Baselines
The naive reference is the stage-1-only detector with KFIoU: a single pass with no refinement cost. Any routed system must beat it in $AP_{75}$ to justify itself. The closest single-pass published reference is the KLD RetinaNet result of 72.39% $AP_{75}$ on HRSC2016 KLD.
The accuracy ceiling is the fixed three-stage KFIoU cascade, run with the same backbone, schedule and training images. This is the strongest competitor on the question asked: it defines what full refinement buys.
As external anchors we report against the strongest published systems in the file. These are 80.90% mAP on DOTA-v1.0 for RoI Trans + KLD + Swin-T MMRotate, 49.41% $AP_{75}$ on DOTA-v1.0 for ARS-DETR with ResNet-50 ARS-DETR, and 90.46 mAP on HRSC2016 for ReDet ReDet. These differ in architecture, so they place our numbers in context rather than serve as matched comparisons. For per-stage refinement latency in oriented cascades, no reported result was found on Lacuna, so we measure it ourselves.
The routing baselines all share the cascade, the routed-fraction budgets and the validation tuning; only the signal used to pick boxes differs. They are classification score; an aspect-ratio threshold; a learned current-overlap predictor in the style of IoU-Net, trained on the same data; and a gain predictor that sees only pooled features. That last one isolates the contribution of the geometric and covariance inputs. A fixed two-stage cascade and IoU-Net-style gradient ascent at matched wall-clock round out the comparison.
Beating the baselines means two things. At a 30–40% routed fraction, the system should be within 0.3 $AP_{75}$ of the full cascade. At every matched fraction, it should beat the current-overlap router by a margin larger than the seed-to-seed spread.
Why not just…?
Why not just train a single-stage detector with KFIoU and skip refinement? Single-pass coupled losses already give large gains. But Cascade R-CNN attributes high-threshold gains to resampling across stages, which one head cannot replicate. The stage-1-only baseline measures exactly what is lost. If the full cascade adds little $AP_{75}$ over it, the routing question is moot, and we would report that.
Why not just run a fixed two-stage cascade? This is cheaper than three stages and, given the saturation reported by Cascade R-CNN and the stage-3 degradation in FSCascade, may capture most of the gain. It appears as a point on our cost axis. Routing must dominate it, delivering more $AP_{75}$ at equal refinement compute, or the method has no reason to exist.
Why not just route on a learned overlap score? This is the real threat. A current-overlap predictor is simple and may implicitly learn which boxes are fixable. Our argument is that low overlap on small objects is often unfixable, while near-threshold elongated boxes are cheaply fixable. The matched-budget comparison, broken down by size and aspect-ratio bins, settles this directly. If the overlap router ties, we will say so: the contribution then reduces to the analysis and the routing system, with the simpler signal recommended.
Why not just a hard aspect-ratio rule? This is free and follows directly from KLD's gradient analysis. It ignores size and confidence, though, and we expect it to over-route clear, easy ships. It is included as a router and can win outright.
Proposed Experiments
The first check is a gate. We profile wall-clock time per image on DOTA-v1.0 tiles and HRSC2016, using a ResNet-50 and a MobileNetV2 backbone. MobileNetV2 matters because R3Det reports up to 23 FPS with it, and light backbones make the refinement heads a larger share of the time. We record the share spent in stages 2–3. Cascade R-CNN reports that added heads cost little. If refinement is under about 10% of total latency even on dense aerial tiles with a light backbone, the latency claim is weak, and we would report the work as compute and analysis results only.
Second, we run the gain analysis comparing Smooth $L_1$ and KFIoU cascades. Third, we plot accuracy against routed fraction for all routers, in $AP_{75}$ and mAP, with milliseconds per image and refinement FLOPs, over three seeds.
Fourth, we test a secondary prediction: that routing near-square boxes out of later stages preserves heading accuracy, given the isotropic-Gaussian failure noted by KFIoU. We measure angle error on square-like categories, comparing routed and fixed cascades.
The decisive outcome is the matched-budget gap between the geometric gain router and the current-overlap router, broken down by aspect ratio.
References
- Learning High-Precision Bounding Box for Rotated Object Detection via Kullback-Leibler Divergence
- The KFIoU Loss for Rotated Object Detection
- ARS-DETR: Aspect Ratio-Sensitive Detection Transformer for Aerial Oriented Object Detection
- Cascade R-CNN: Delving Into High Quality Object Detection
- Rethinking Classification and Localization for Cascade R-CNN
- R3Det: Refined Single-Stage Detector with Feature Refinement for Rotating Object
- Acquisition of Localization Confidence for Accurate Object Detection
- iRe2f: Rethinking Effective Refinement in Language Structure Prediction via Efficient Iterative Retrospecting and Reasoning
- Meta-Reflection: A Feedback-Free Reflection Learning Framework
- MMRotate: A Rotated Object Detection Benchmark using Pytorch
- Gaussian YOLOv3: An Accurate and Fast Object Detector Using Localization Uncertainty for Autonomous Driving
- BAR - A Reinforcement Learning Agent for Bounding-Box Automated Refinement
- ReDet: A Rotation-Equivariant Detector for Aerial Object Detection