Object detectors increasingly improve their boxes in several passes: a first guess, then one or more correction steps, each trained to hit the ground truth more tightly. These extra passes cost time, and their measured benefit is concentrated in the strictest overlap metrics. Those metrics assume the human-drawn ground-truth box is exact. It is not. Annotators disagree by several pixels, occlusion makes edges ambiguous, and this kind of spatial noise is hard to filter because it blends smoothly into clean data. Once a predicted box sits within the annotation's own noise band, a further correction cannot be verified by any evaluation. It may also be teaching the model to copy annotator jitter.
This note proposes a stopping rule for multi-stage box refinement. At each stage, the detector predicts two things: the size of the correction the next stage would make, and the spread of the annotation noise on each box side. Refinement stops, box by box, when the predicted correction is smaller than that spread. The same quantity supports an instance-level evaluation that only credits a refinement step when it moves a box by more than label noise could explain. The central prediction separates this rule from generic confidence-based early exit. As label noise rises, the rule should exit earlier on its own, saving more computation at no loss in accuracy against cleaner references. A fixed confidence threshold would instead need to be retuned for every noise level.
Where the four directions meet
Coordinate regression, meaning direct prediction of continuous box coordinates, is unstable. Targets vary over wide ranges and outputs are unbounded (PointFusion). Predicting corrections relative to a coarse estimate avoids local minima that trap one-shot regressors (Residual Pose).
The most successful version in detection is Cascade R-CNN. It stacks three refinement heads on a shared feature extractor and trains each at a stricter overlap threshold (intersection-over-union, or IoU, of 0.5, 0.6 and 0.7). This raises the strictest-threshold score, AP at IoU 0.90, from 7.1 to 15.8 on COCO. Side-Aware Boundary Localization (SABL) attacks the same instability differently. It classifies each box edge into a coarse bin, regresses a small offset within the bin, and adds about 10% inference overhead. Both papers motivate this proposal, because their gains live exactly where annotation precision matters most. Both also leave its premise unexamined: they train and evaluate as if ground-truth edges were exact, and every proposal passes through every stage.
Two papers say the premise is false and also supply our methods. Bounding Box Regression With Uncertainty shows that COCO and PASCAL VOC boxes are ambiguous. It trains each box side as a Gaussian with a mean and a spread $\sigma$ using a KL-divergence loss, so the model learns a large $\sigma$ on ambiguous edges. It then fuses overlapping candidates weighted by their predicted spread, a step it calls variance voting. We reuse its per-side Gaussian head as our noise estimator. The paper never uses $\sigma$ to decide how much refinement an instance deserves, and it evaluates only with dataset-level average precision. Benchmarking Label Noise in Instance Segmentation builds COCO-N and related benchmarks with controlled localization, scale and simplification noise. Under hard noise, Mask R-CNN falls from 34.6 to 22.3 mAP. The paper also finds that standard noisy-label filters miss spatial noise. That finding constrains our design: we cannot expect to clean the noise away first, so the model must reason about it.
Two lines of work shape the evaluation and latency framing. The instance-level evaluation literature warns that dataset averages hide instance failures (Adding Chocolate to Mint) and that local scores catch errors global ones miss (Simple Token-Level Confidence). Re-TACRED shows that label noise can erase real differences between architectures, not just shift every score by a constant. On latency, Meta-Reflection collapses multi-pass self-correction in language models into one pass. BAR refines boxes with a reinforcement-learning agent whose reward is IoU against a reference that may itself be noisy. Neither asks when a spatial correction stops being meaningful, and that is the question this proposal addresses.
Core Claim
A box correction smaller than the annotation noise on that box can neither be verified at test time nor learned reliably at training time. A cascade should therefore spend its later stages only on instances where the expected correction exceeds the noise.
Let stage $k$ output, for each box side $s$, a Gaussian with mean $\mu_{k,s}$ and spread $\sigma_{k,s}$. Let it also output $\hat{\Delta}_{k+1,s}$, a prediction of the offset that stage $k+1$ would apply to side $s$. The box exits after stage $k$ if
$$ \max_{s} \frac{|\hat{\Delta}_{k+1,s}|}{\sigma_{k,s}} < \lambda , $$
where $\lambda$ is a single scalar tuned once on clean validation data and then held fixed across all noise levels.
The correction predictor is a small head trained to match the next stage's actual offsets, so its targets come from the cascade itself. Because $\sigma$ is trained with the KL loss, it widens when training labels are noisier. That widening is what should make exits rise automatically with noise.
Research Program
The work has three parts.
The first part is the model. We take a three-stage Cascade R-CNN with a ResNet-50 feature pyramid. We replace each stage's box regressor with the Gaussian head and KL loss of He et al., and add the correction-predictor head. We will also build the same model on SABL-style edge bins, with $\sigma$ predicted on the within-bin offset. This tests whether the rule depends on the regression parameterization.
The second part is a controlled noise testbed. We derive boxes from COCO-N's noisy masks at several noise strengths and also inject direct box-edge jitter and scale erosion. This gives training and test labels whose noise distribution we know.
The third part is an instance-level evaluation. For each test instance we draw many independent noisy versions of its reference box from the known noise model. This yields the distribution of IoU changes that label noise alone produces. A refinement step on that instance counts as a verified improvement only if its IoU gain against the original COCO box exceeds the 95th percentile of that noise-only distribution. Verified regressions are defined symmetrically. The original COCO boxes are themselves imperfect, so we treat them as cleaner than the injected-noise labels, not as truth, and we state this limitation plainly.
Baselines
The naive reference point is a single-stage Faster R-CNN regressor with Smooth L1 loss, trained on the same noisy labels. This anchors how much any refinement buys.
The strongest existing method is the full three-stage Cascade R-CNN, run on every proposal. On clean COCO it reports 38.9 AP and 15.8 AP90 with an FPN+ backbone (Cascade R-CNN). SABL reports 43.3 AP on test-dev when added to Cascade R-CNN (SABL). We found no reported result on Lacuna for any cascade detector trained on COCO-N or other spatially noisy box labels, so we will reproduce these baselines ourselves under every noise level.
A third baseline is the cascade with KL loss and variance voting but no early exit, following He et al., who report 29.1 AP for VGG-16 Faster R-CNN with both components. This separates the benefit of modeling noise from the benefit of skipping stages.
The key ablation is a learned early exit on a fixed confidence signal. Each box exits when its predicted IoU, or its classification score, passes a threshold. That threshold is tuned on clean validation data to match our method's average number of stages.
All baselines share the backbone, the training schedule, the hyperparameter search budget and the validation data used for tuning. Timing uses the same GPU and batch size, and we report both per-proposal head compute and end-to-end wall-clock time.
Beating the full cascade means matching its AP and AP75 against the original COCO boxes, within seed variance, while using measurably fewer stage evaluations per proposal. Beating the fixed-threshold exit means a better accuracy-compute trade-off at noise levels it was not tuned on.
Why not just…?
Why not just early-exit on predicted IoU or classification confidence? This is the real threat. If a predicted-IoU threshold tuned once performs as well at every noise level, our contribution shrinks to "use uncertainty for early exit." The two rules make different predictions. A fixed IoU target keeps demanding the same tightness as labels get noisier, so it should spend the same or more compute on refinements that cannot be verified. Our rule should exit earlier as noise rises. The noise sweep in the experiments tests exactly this. If the curves coincide, we will report it as a negative result.
Why not just train fewer stages, or a cascade with KL loss and no exit? Dropping the third stage everywhere loses the AP90 gains on instances that are genuinely sharp, such as isolated rigid objects. Keeping all stages pays full cost on ambiguous ones. The per-instance verdicts show whether the third stage's verified improvements concentrate on low-$\sigma$ instances. If they do not, a uniform two-stage cascade is the right answer.
Why not just clean the labels with noisy-label filters or SAM? Grad et al. report that standard filters miss spatial noise. They also report that training on SAM-generated labels yields 25.3 mAP versus 34.6 on human labels (Benchmarking Label Noise). Cleaning with a foundation model therefore trades edge jitter for semantic errors rather than removing noise.
Why not just report AP50 and AP75 under noise and skip instance verdicts? Looser thresholds hide exactly the refinements at issue. Re-TACRED shows that aggregate scores on noisy labels can merge genuinely different systems (Re-TACRED). We will test directly whether the ranking of cascade variants by AP90 on noisy test labels agrees with their ranking by verified instance improvements.
Proposed Experiments
The decisive experiment is a noise sweep. We train every model at five noise strengths, from clean to COCO-N "hard". For each, we plot accuracy against the original COCO boxes (AP, AP75, AP90) against mean stages per proposal and wall-clock time.
The method succeeds if three things hold together. Its exit rate rises with noise. Its accuracy against the cleaner reference stays within seed variance of the full cascade. The fixed-threshold exit either loses accuracy or saves less computation at the noise levels it was not tuned on.
A second check asks whether the learned $\sigma$ tracks the injected noise. We regress per-instance $\sigma$ on known injected noise magnitude. If $\sigma$ does not track it, the stopping rule has no basis.
A third check is the ranking test. We compare AP90 on noisy test labels with verified-improvement counts across the five cascade variants and report rank agreement.
We also repeat the main comparison on PASCAL VOC with injected jitter and on Cityscapes-N boxes, to check that the effects transfer beyond COCO. Honest expectations on latency matter here: the shared feature extractor dominates runtime, so wall-clock savings are bounded by the share of time spent in the heads. We will report that share explicitly rather than only stage counts.
References
- PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation
- Residual Pose: A Decoupled Approach for Depth-based 3D Human Pose Estimation
- Cascade R-CNN: Delving Into High Quality Object Detection
- Side-Aware Boundary Localization for More Precise Object Detection
- Bounding Box Regression With Uncertainty for Accurate Object Detection
- Benchmarking Label Noise in Instance Segmentation: Spatial Noise Matters
- Introduction to China's CWMT2008 Machine Translation Evaluation
- Simple Token-Level Confidence Improves Caption Correctness
- Re-TACRED: Addressing Shortcomings of the TACRED Dataset
- Meta-Reflection: A Feedback-Free Reflection Learning Framework
- BAR - A Reinforcement Learning Agent for Bounding-Box Automated Refinement