Rank-Stable Detector Ensembles Under Distance Degradation

Adversarial patches printed on clothes, vehicles, or roadside objects fool object detectors in the lab but quietly stop working as the target moves away from the camera. The reason is physical: as distance grows, each camera pixel integrates light from a larger patch of the world, which acts as a low-pass filter that blurs the high-frequency adversarial texture back into harmless background. This "distance-precision trade-off" is now well documented — white-box patches that crush Faster R-CNN accuracy from 95.03% to 17.46% at 50 m altitude lose more than half their potency by 90 m, where accuracy rebounds to 51.10% on the EVD4UAV benchmark EVD4UAV: An Altitude-Sensitive Benchmark to Evade Vehicle Detection in UAV. The natural defensive response is to run several detectors and fuse their outputs, on the theory that a two-stage anchor-based detector, a single-stage detector, and a transformer detector will not all fail together.

I propose to test and then fix a specific weakness in that response. My core hypothesis is that distance does not merely weaken attacks — it collapses ensemble diversity. As the optical low-pass filter strips away architectural nuance, structurally different detectors are pushed into a shared, correlated failure mode, so the ensemble's supposed redundancy evaporates exactly when the attack is strongest. The fix I will build is a fusion rule that aggregates detectors by rank rather than by raw confidence scores, trained end-to-end with a differentiable ranking surrogate. Rank-based aggregation is invariant to the absolute scale of each detector's scores, so a detector whose confidences collapse or skew under blur cannot dominate the consensus, and the ensemble keeps the failure independence that score averaging destroys.

Core Claim

The claim has two halves, one diagnostic and one constructive. First, the error correlation between detectors in a vision ensemble is not a constant but a function of viewing distance: I expect pairwise error correlation to rise steeply with distance under physical attack, because the shared low-pass filtering of the physical channel dominates the architectural differences that normally decorrelate detector errors. This has never been measured. The ensemble-redundancy literature has quantified correlated failures across models — error correlations frequently exceeding 0.7 across algorithmic families Quantifying Correlations of Machine Learning Models, agreement on the same wrong answer roughly 60% of the time among top language models Kim et al., and pre-training pinning structurally distinct networks into the same optimization basin Neyshabur et al. — but always at a fixed input distribution, never as a trajectory along a physical degradation axis. The closest work on the vision side, Full-Distance Evasion of Pedestrian Detectors in the Physical World and EVD4UAV, evaluates multiple detectors only side by side, never as a fused system, and never measures whether their failures coincide.

Second, if the correlation collapse is real, the remedy is to change what the ensemble agrees on. Existing multi-detector aggregation — confidence averaging, box voting, non-maximum suppression — operates on raw scores whose scales drift apart under distance blur, and the score-fusion failure mode is well characterized: Rethinking Multi-objective Ranking Ensemble in Recommender System: From Score Fusion to Rank Consistency showed that score-level fusion degrades by nearly 10% under 100× label skew while a rank-consistent differentiable formulation degrades by only 1.3%. I will port that insight from tabular recommender logs, where it currently lives, to spatial proposal lists under physical attack. The mechanism is a fusion head that replaces each detector's raw proposal scores with their soft ranks before averaging, then trains the whole ensemble-on-top with a smooth ranking loss in the spirit of Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval.

Research Program

The program runs in three stages, each feeding the next.

Stage 1: Measure redundancy collapse along the distance axis. I will build the first "correlation-versus-distance" curves for detector ensembles. Using EVD4UAV EVD4UAV: An Altitude-Sensitive Benchmark to Evade Vehicle Detection in UAV, which provides 6,284 aerial images with 90,886 vehicle annotations at 50 m, 70 m, and 90 m under three lighting conditions, I will run Faster R-CNN, DETR, and YOLOv8 — the three architectures the benchmark already covers — both clean and under its white-box and black-box patch attacks, and compute pairwise error correlation (Pearson's r and the non-linear $\phi_K$ coefficient used by Sarna et al.) and the $Q$-statistic (used by Abels and Lenaerts, who found $Q \approx 0.85$ for correlated LLM ensembles versus 0.38 for genuinely diverse human crowds) as continuous functions of altitude. I will interpolate intermediate altitudes synthetically using the Distant Image Converter from Cheng et al., whose physically-grounded degradation model (atmospheric perspective, lens anti-aliasing, sensor blur, ISP effects) holds reconstruction error stable at an L2 norm of about 0.11 across distances — far better than naive downsampling, which Cheng et al. show produces a systematic appearance gap between digital simulation and real capture. The deliverable is an empirical law: at what distance, and under which attack regime, does cross-architecture error correlation exceed the threshold where voting amplifies rather than cancels errors.

Stage 2: Build the rank-consistent fusion head. For a set of $M$ detectors, each producing scores $s_i^{(m)}$ over a pooled proposal list, I compute each detector's soft rank for proposal $i$ using the sigmoid surrogate from Brown et al.:

$$r_i^{(m)} = 1 + \sum_{j \neq i} \sigma\!\left(\frac{s_j^{(m)} - s_i^{(m)}}{\tau}\right)$$

where $\tau$ is a temperature controlling the smoothing. The consensus score is a learned, per-image weighted combination $c_i = \sum_m \alpha_m r_i^{(m)}$, with attention weights $\alpha_m$ conditioned on the estimated viewing distance (available in EVD4UAV metadata and from the DIC simulator) and trained end-to-end with a Smooth-AP-style loss on the consensus ranking against ground truth. This mirrors the commonality-modeling-plus-personalized-query design of Xia et al., but over bounding-box proposal lists rather than item lists, and without their e-commerce objective heads. The key property is scale invariance: ranks are unchanged under any monotone rescaling of a detector's scores, so a detector whose confidences sag or saturate under blur contributes a consistent ordering rather than a dragged-down average. I will compare the sigmoid surrogate against the Sinkhorn optimal-transport relaxation of Cuturi, Teboul, and Vert, which sorts by entropically-regularized assignment and gives exact gradients in $O(nm)$, and against the identity-update formulation of Rank & Sort Loss for Object Detection and Instance Segmentation, which already proved that rank supervision can jointly rank positives above negatives and sort positives by localization quality in a single detector — I extend that principle from intra-model training to inter-model fusion under attack.

Stage 3: Close the loop on the attack side. A defense is only meaningful against the strongest attack, so I will re-optimize attacks against the fused ensemble. Cheng et al. attack ensembles by uniform averaging of losses across surrogate detectors, achieving over 75% attack success rate on black-box detectors; Duan et al. attack the top-$n$ proposals of a Region Proposal Network simultaneously to transfer across architectures. I will run both attack styles against the rank-fused ensemble, including attacks that explicitly target the fusion weights $\alpha_m$, to test whether rank aggregation raises the cost of ensemble-breaking rather than merely shifting the attack surface.

Baselines

The naive reference point is the best single detector with no fusion. On EVD4UAV, that is YOLOv8, which reports 98.8% mAP50 on the clean benchmark Sun et al.; under white-box patch attack at 50 m, the comparable single-model figure is Faster R-CNN's collapse from 95.03% to 17.46% accuracy, recovering to 51.10% at 90 m. Any fusion is only worth building if it beats these single-model numbers under attack at every altitude, not just on average.

The strongest existing competitor is the score-averaged multi-detector ensemble with standard non-maximum-suppression fusion, exactly as constructed for ensemble transferability attacks by Cheng et al., who average detector losses uniformly across surrogates, and the same three architectures used in EVD4UAV combined by confidence-weighted box fusion. This is the strongest competitor because it is the default practice, it uses the same underlying detectors as my method, and Stage 1 is explicitly designed to characterize when it fails. To keep the comparison fair, the score-averaged ensemble, a weighted-box-fusion ensemble, and my rank-fused head will all use identical detector checkpoints, identical training images and altitude sampling, identical tuning budget (the same random-search allowance over fusion hyperparameters), and identical evaluation protocol. The score-ensemble baseline gets the same DIC-simulated multi-distance training data so that any gap is attributable to the fusion rule, not to data. Beating it means: higher mAP50 and detection accuracy under both white-box and black-box patch attacks at 50 m, 70 m, and 90 m, with the margin growing as the Stage-1 correlation curve rises — if rank fusion wins only where correlation is low, the mechanism claim is refuted.

Two ablations isolate the key ingredient. The first replaces the soft-rank transformation with raw scores inside the same learned attention fusion head, holding architecture, data, and tuning fixed; the gap between this ablation and the full method is the contribution of rank-based scale invariance alone. The second swaps the sigmoid surrogate for the Sinkhorn relaxation of Cuturi et al. to test whether the specific relaxation matters or only rank-consistency per se. A third ablation disables the distance conditioning of the fusion weights $\alpha_m$, testing whether adapting aggregation to the degradation regime matters or whether a static rank fusion suffices.

Why not just…?

Why not just train with expectation over transformation and average scores? This is the standard recipe: sample many synthetic scales and blurs during optimization or adversarial training, and average the detector losses. The evidence against it is direct. Cheng et al. showed that scalar loss summation across distances creates genuine optimization conflicts — the high-frequency components needed at short range cancel the low-frequency components needed at long range — requiring an elaborate two-stage schedule, and that naive downsampling fails to reproduce physical optics at all (their physically-motivated predecessor baselines achieved only 22% attack success rate where their optics-aware method achieved 74%). On the defense side, Oksuz et al. showed that scalar classification losses misalign confidence with localization quality, while rank supervision fixed this without hyperparameter balancing, and Xia et al. showed score averaging collapses by nearly 10% under skew where rank formulations drop 1.3%. Still, I will run the EOT-plus-score-averaging baseline head-to-head under identical simulation budgets, because if it matches rank fusion under a physically-grounded simulator, the extra machinery is not justified.

Why not just ensemble architecturally diverse detectors and vote? Faster R-CNN, DETR, and YOLOv8 differ in almost every design choice, so one might assume their failures are independent by construction. The redundancy literature says otherwise: Sarna et al. found error correlations above 0.7 even between random forests and neural networks when trained on the same data; Kim et al. found that stronger models agree on the same wrong answers about 60% of the time; Abels and Lenaerts found that majority voting among correlated voters actively amplifies the consensus error ($Q \approx 0.85$). And Neyshabur et al. showed shared pre-training pulls structurally different networks into one representation basin. My Stage 1 is precisely the experiment that settles this for the distance axis: if voting ensembles track the best single detector rather than exceeding it as correlation rises, naive diversity is dead; if they hold up, my proposal loses its motivation and I will say so.

Why not just use weighted box fusion as a post-processor and skip the differentiable machinery? Weighted box fusion and confidence-weighted voting are cheap, training-free, and standard. But they are score-scale operators, and the entire premise of this proposal is that distance blur warps score scales non-uniformly across detectors. Xia et al. demonstrated the score-versus-rank gap empirically in the skew setting, and the analogy is close enough to take seriously. The honest answer is that this alternative is a real threat if score distortion turns out mild, and the Stage 1 measurements of per-detector score-distribution drift as a function of distance will show whether rank invariance is actually needed — this is why weighted box fusion is included as a full baseline rather than dismissed.

Why not worry about the computational cost of differentiable ranking? Pairwise soft ranking is quadratic in the proposal count, and Sinkhorn iterations add their own constant factor; Oksuz et al. report their sorting loss increased training time about 1.5×. But the overhead is bounded and known: Brown et al. added under 3 ms per training iteration with their sigmoid surrogate, and Xia et al. deployed a fast rank-sum formulation at $O(n \log n)$ in a production system with 4 ms latency serving 400 million daily users. I will restrict soft ranking to the pooled top-$k$ proposals per image (a few hundred), report wall-clock overhead explicitly, and prefer the linear-time rank-sum form if the Sinkhorn variant proves slow.

Proposed Experiments

The decisive checks, in order of importance. First, the correlation law: on EVD4UAV at 50/70/90 m with interpolated intermediate altitudes from the DIC simulator, plot cross-detector error correlation and $Q$-statistic versus distance, clean and under attack. The hypothesis predicts correlation rising with distance and rising fastest under attack; a flat curve falsifies the premise of the whole proposal. Second, the fusion comparison: under white-box patches (the regime where single-model accuracy fell from 95.03% to 17.46% at 50 m and recovered to 51.10% at 90 m Sun et al.), the rank-fused ensemble should hold detection accuracy above every single model and above the score-averaged ensemble at each altitude, with the margin over score averaging growing in proportion to the measured correlation curve. Third, the attack-side stress test: re-optimize patches against the fusion itself, using the ensemble-attack recipe of Cheng et al. — whose attacks achieved 74% success (patches) and 76% (clothing) against YOLOv5 and over 75% against black-box detectors via ensemble loss averaging — and the proposal-targeting strategy of Duan et al., whose coated camouflage dropped source-detector AP from 1.00 to 0.06. The defense succeeds if the attack success rate against the rank-fused ensemble is materially lower than against the score-averaged ensemble built from the same detectors. Fourth, the physical check: printed patch attacks on a small physical capture campaign following the distance protocol of Cheng et al., from roughly 4 m to 40 m, to confirm the simulation results survive the appearance gap between digital downsampling and real optics. For the ranking surrogate itself, a sanity benchmark on Stanford Online Products — where Brown et al. report 80.1% Recall@1, outperforming their baseline by 1.5% — will verify that my pooled-proposal implementation of the surrogate matches its published behavior before it is trusted under attack.

The throughline is simple to state: distance is a physical channel that all detectors share, shared channels create shared failures, shared failures break score-level ensembling, and rank-level ensembling is the one aggregation that provably does not care about score scales. The proposal tests that chain link by link, and each link has a pre-registered way to fail.