An object detector does two jobs at once. It says what an object is (classification), and it draws a box around it (localization). In aerial imagery that box is rotated, so it has an angle as well as a center and size. A simple training trick from image classification, called MaxUp, generates several augmented copies of each training image and trains only on the copy the model finds hardest. This acts as a cheap form of adversarial training. Carrying the trick over to oriented detection raises two problems. First, "hardest" is measured by one summed loss, so whichever task currently has the larger loss decides which copy is chosen, and the other task gets trained on copies that may be easy for it. Second, some hard copies are hard because they are broken: the object is cut off at the patch border, or a small vehicle is blurred to a few pixels. A model trained on those copies is fitting labels it cannot recover.
We propose to pick hard augmentations separately for each task, and to measure regression hardness as surprise rather than raw error. Surprise here means error divided by the model's own predicted uncertainty. A copy where the model is confidently wrong about a ship's angle is worth training on. A copy where the error is large but the model already reports high uncertainty, such as a truncated or blurred object, probably carries little new information. The detector learns this uncertainty with a standard Gaussian regression loss, so the same quantity serves two purposes: it weights the regression loss, and it acts as the filter that keeps adversarial selection from chasing corrupted copies.
Core Claim
In a two-stage oriented detector, two changes to worst-case augmentation selection should give higher accuracy at strict box-overlap thresholds than either uniform random augmentation or plain MaxUp, at matched compute. The first change is to select the classification candidate and the regression candidate independently. The second is to rank regression candidates by uncertainty-normalized error rather than raw loss. The gain should be concentrated on slender, high-aspect-ratio classes such as ships, bridges and large vehicles, where a degree or two of angle error destroys overlap.
Each piece has precedent but the combination does not. MaxUp supplies the selection mechanism. It takes the maximum-loss candidate from a pool and shows this behaves like a gradient-norm penalty. It is defined for one scalar loss and was tested only on classification and certified robustness. Glitter and Hard View Pretraining make the same single-objective, max-loss choice, in other settings. EntProp links uncertainty to augmentation by sending high-entropy samples through extra augmentation. However, it uses uncertainty to seek hardness, not to discount it, and it stays within image classification. ADDMU is the closest in spirit. It finds that adversarial text examples with high model uncertainty but low data uncertainty are the informative ones. That is the same separation we want between "confidently wrong" and "inherently ambiguous", but ADDMU applies it to detecting attacks on text classifiers, not to selecting training augmentations for spatial regression.
Research Program
We build on Oriented R-CNN, which produces oriented proposals by predicting offsets from the midpoints of a horizontal box's edges, then classifies and refines them using features extracted from rotated regions. Its second-stage regression branch is changed in the style of He et al.'s KL loss. The branch predicts a mean $\mu_d$ and a log-variance $\log\sigma_d^2$ for each box parameter $d$, and is trained with the loss $|t_d-\mu_d|/\sigma_d^2+\tfrac12\log\sigma_d^2$, where $t_d$ is the target value. He et al. use this loss on horizontal boxes only, with random augmentation. We extend it to the oriented parameters and use the predicted variance for candidate selection as well as for loss weighting.
For each training patch we draw $K$ candidates (default $K=4$) from a geometric and photometric pool. The pool includes arbitrary-angle rotation with exact box rotation, scale changes, flips, crop placement and color jitter. Following MaxUp's low-resolution acceleration, all candidates are scored in a forward pass at reduced resolution. For candidate $k$ we compute two scores over its positive (object-matched) samples $j$:
$$ s_{\text{cls}}(k)=\frac{1}{|P_k|}\sum_{j\in P_k}-\log p_{j,y_j},\qquad s_{\text{reg}}(k)=\frac{1}{|P_k|\,D}\sum_{j\in P_k}\sum_{d=1}^{D}\frac{|t_{jd}-\mu_{jd}|}{\sigma_{jd}}. $$
Here $P_k$ is the set of positive samples in candidate $k$, $p_{j,y_j}$ is the predicted probability of the true class, and $D$ is the number of box parameters. We set $k_{\text{cls}}=\arg\max_k s_{\text{cls}}(k)$ and $k_{\text{reg}}=\arg\max_k s_{\text{reg}}(k)$. The update applies the classification loss on candidate $k_{\text{cls}}$ and the KL regression loss on candidate $k_{\text{reg}}$, using masked losses in one combined backward pass. When the two indices coincide, which we will measure, only one copy is used.
Ranking within the pool makes the regression score robust to a model that simply inflates all its variances, because a uniform rescaling of $\sigma$ does not change which candidate ranks first. The design is also deliberately asymmetric. Cross-entropy is already a surprise measure for classification, so classification gets no learned variance. Reviewers may question this choice, and the ablation below tests it.
The analysis has three parts, each answering one question. The first asks whether joint-loss MaxUp really lets classification dominate the choice. We log how often $k_{\text{cls}}\neq k_{\text{reg}}$, and which task's preferred candidate the joint loss selects. The second asks whether surprise-based selection avoids corrupted candidates. We record the visible fraction of each object after cropping, and compare the truncation rate of candidates chosen by raw-loss versus surprise selection. The third asks whether the variance remains meaningful under adversarial training. We check whether predicted $\sigma$ correlates with actual test-time overlap error, and whether variance-weighted box merging during duplicate removal, as in He et al., still helps.
As a secondary, clearly optional extension, we plan to replace the independent per-parameter variances with a covariance that couples angle and aspect ratio. One way is to express box uncertainty through a Gaussian box representation like Linear Gaussian bounding box representation. This targets a known weakness: angle error on a long object moves its corners far more than on a square one. The main claim does not depend on this extension.
Baselines
The naive reference point is Oriented R-CNN with a ResNet-50 backbone trained with standard random augmentation. Oriented R-CNN reports 75.87% mAP on DOTA and 96.50% mAP on HRSC2016 for this configuration. We will reproduce it in our own codebase and compare against our reproduced number, not the published one, so that implementation differences do not count in our favor. It is the strongest relevant competitor because it is the detector we modify, and it already performs well on both benchmarks.
The compute-matched random baseline uses the same detector with the same augmentation pool, sampled uniformly. It gets extra training iterations or a larger batch so that its total forward and backward compute equals our method's. It can see more than four times as many augmented images. Beating it at equal GPU-hours is the minimum bar.
Joint MaxUp applies MaxUp directly, selecting the candidate with the largest summed detection loss. It uses the same pool, the same $K$ and the same low-resolution scoring. This is the most direct competitor for the selection idea.
The uncertainty-only baseline is Oriented R-CNN with the KL regression head and random augmentation, plus variance-weighted box merging. It isolates how much gain comes from uncertainty modeling alone. On horizontal COCO boxes, He et al. report an improvement in Faster R-CNN (VGG-16) AP from 23.6% to 29.1%. For oriented boxes on DOTA, we found no reported result on Lacuna for KL loss on Oriented R-CNN.
The key ablation is a two-by-two design that crosses selection criterion (raw loss versus surprise) with candidate choice (one joint candidate versus task-split candidates). All four cells use the KL head. The proposal beats the baselines if the surprise, task-split cell exceeds both joint MaxUp and the compute-matched random baseline. The required margin is at least the seed-to-seed spread across three seeds, on DOTA mAP and on a strict-threshold metric (AP at IoU 0.75). It must also show the larger gain on slender classes. If the improvement appears only in the cell that uses both changes, the claim that the two changes interact is supported.
Why not just…?
Why not just use random augmentation with a longer schedule? This is the most serious threat. Rotation augmentation in overhead imagery is nearly label-preserving and cheap, so seeing many more random copies may match careful selection. Evidence from classification points the other way. MaxUp improved ResNet-50 CutMix accuracy on ImageNet from 78.6% to 78.9%, and EntProp raised CIFAR-100 $H_{score}$ from 62.08 to 69.12 through targeted selection. These gains are modest, though, and come from a different task. The compute-matched random baseline settles the question directly. If it ties us, we will report that selection is not worth its cost in this setting.
Why not just apply MaxUp to the summed detection loss? It is simpler and needs no uncertainty head. Our concern is that classification cross-entropy dominates the sum, so regression trains on candidates chosen for the wrong reason. That concern is a hypothesis drawn from TSD and DDOD, which show that the two tasks prefer different features and different sample assignments. It has not been measured for augmentation selection. The disagreement rate between $k_{\text{cls}}$ and $k_{\text{reg}}$ tests it directly. If the two indices nearly always coincide, task splitting adds nothing and the claim narrows to surprise-based scoring.
Why not just use the KL uncertainty loss with ordinary augmentation? Uncertainty weighting already discounts ambiguous boxes, so selection might be redundant. We expect a different interaction. Raw-loss selection preferentially picks truncated and blurred copies, which the KL loss then down-weights. That cancels much of the benefit of selecting hard copies, which is the attenuation-versus-mining tension in the research file. Surprise selection avoids this by picking copies the model is confidently wrong about. The uncertainty-only baseline and the raw-loss/surprise contrast together test this. Comparing truncation rates of selected candidates shows whether the mechanism works as described.
Why not just mine hard regions within each image instead of augmenting? Mining over proposals from an unchanged image reweights examples the model already sees. It does not expose the model to new orientations or scales of the same object, and it lacks MaxUp's link to gradient-norm smoothing. We will not run full online hard example mining as a headline baseline, because the uncertainty-only baseline covers per-instance reweighting. This is an acknowledged gap in the comparison set.
Proposed Experiments
The main experiments train on DOTA using the standard 1024×1024 overlapping patches, report mAP over its 15 classes, and give per-class AP for ship, bridge, large vehicle and harbor. HRSC2016 is a second benchmark. Its long ships are where the angle-sensitivity argument should be strongest, but the baseline is already near its ceiling at 96.50% mAP, so we will also report AP at stricter overlap thresholds. FAIR1M serves as a fine-grained transfer check. We found no reported result on Lacuna for Oriented R-CNN on FAIR1M, so there we compare only against our own reproduction. All runs use three seeds, identical pools and the same tuning budget per method. Results report wall-clock training time alongside accuracy.
Three checks decide the outcome. The first is whether the full method beats the compute-matched random baseline on DOTA at IoU 0.75. If it does not, the idea fails. The second is whether, among candidates selected by raw loss and by surprise, the surprise-selected ones contain measurably fewer heavily truncated objects. This tests the claimed mechanism, not just the final score. The third is a deliberate stress test. We inject a known fraction of corrupted candidates, such as crops that remove most of an object, and check whether raw-loss selection degrades and surprise selection does not. Together these show whether uncertainty can serve as the guard that makes worst-case augmentation safe for oriented detectors trained on two tasks at once.
References
- MaxUp: Lightweight Adversarial Training With Data Augmentation Improves Neural Network Training
- When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentation
- Beyond Random Augmentations: Pretraining with Hard Views
- EntProp: High Entropy Propagation for Improving Accuracy and Robustness
- ADDMU: Detection of Far-Boundary Adversarial Examples with Data and Model Uncertainty Estimation
- Oriented R-CNN for Object Detection
- Bounding Box Regression With Uncertainty for Accurate Object Detection
- Linear Gaussian bounding box representation and ring-shaped rotated convolution for oriented object detection
- Revisiting the Sibling Head in Object Detector
- Disentangle Your Dense Object Detector