Modern transformer object detectors in the DETR family draw a box through a fixed pipeline. Each candidate object, called a query, is passed through a stack of typically six decoder layers, and each layer nudges the box a little closer to the object. Every query pays for every layer: a box that settled at layer two costs the same as one still drifting at layer six, and so does a query that sits on empty background. This proposal asks whether a query's hidden activations already show that its box is finished. We test this with a tiny classifier, a linear probe, that reads the query's internal vector at an intermediate layer. If the probe can tell, finished queries can stop refining early and the saved decoder compute goes to the queries that still need it.
The bet borrows from language-model uncertainty work, where cheap probes on hidden states predict when a model is about to be wrong more reliably than the model's own output confidence does. The novelty is in what the probe predicts. It does not estimate how good the box is now, which is what every existing uncertainty-aware detector does. It predicts how much the remaining layers will still change the box. That quantity decides whether more computation is worth spending. It can also be supervised with no human labels, using the frozen detector's own final-layer boxes. The key test is whether the hidden state carries this signal beyond what is already visible in the outputs, such as the size of the last box update or the class score. If it does not, the right conclusion is that simple heuristics suffice, and the experiments are built to report that honestly.
Core Claim
For a query $q$ at decoder layer $l$, define the remaining refinement as $r_{q,l} = 1 - \mathrm{IoU}(b_{q,l}, b_{q,L})$. Here $b_{q,l}$ is the query's box after layer $l$, $b_{q,L}$ is its box after the last layer $L$, and IoU is intersection over union. We claim three things. First, a linear probe on the query hidden state $h_{q,l}$ predicts $r_{q,l}$, and predicts whether the query ends up matched to any object, better than a probe of the same size built only on observable output features. Those output features are the class score, the magnitude of the last box update, and, for D-FINE, the entropy of its boundary distributions. Second, the probe signal becomes reliable at a specific, measurable decoder depth. Third, gating queries on this signal cuts decoder compute substantially at matched AP. The savings should be largest in dense scenes, where query counts and decoder cost are highest.
Relation to Prior Work
The motivating evidence comes from language models. Semantic Entropy Probes train logistic-regression probes on LLM hidden states to predict semantic uncertainty from a single forward pass. They report a 10.5-point AUROC gain over accuracy probes on new tasks with Mistral-7B, and the signal already appears before generation begins. Learning Confidence for Transformer-based NMT trains an auxiliary confidence network on lower decoder layers of a translation model and finds those layers less biased than the top ones. Together these two papers disagree about which depth to read. That disagreement is why we map probe quality layer by layer instead of assuming a location. Both works use uncertainty passively, to detect errors or smooth labels. We use it to control computation.
The constraining and closest work sits in detection. Bounding Box Regression With Uncertainty adds a per-edge Gaussian variance head to two-stage CNN detectors trained with a KL loss. UADET turns similar variance outputs into attention maps for the dense detector FCOS. An Uncertainty-aware DETR Enhancement Framework models DETR boxes as Gaussians and feeds a Bayes-risk term into embeddings, losses and matching. It gains +1.2 AP on DINO but adds about 4% inference time. All three retrain the detector with explicit output uncertainty heads, all three estimate current localization error, and none uses uncertainty to reduce computation.
D-FINE comes closest in spirit. It refines boundary distributions layer by layer and distills deep-layer boxes into shallow layers (GO-LSD). Its step sizes follow a fixed weighting function, and every query still runs every layer. We reuse its insight that the final layer is a good teacher for earlier ones. We use that teacher to build targets for stopping decisions rather than distillation losses.
DQ-DETR adapts compute by choosing among 300 and 1500 queries per image from a global density estimate. That choice is image-level and made once. The authors observe that it trails DINO on COCO (50.2 vs 51.3 mAP), a hint that coarse allocation is not enough. Our gating is per query and per layer. DEED does early exit for encoder-decoder language transformers. It exits whole decoding steps, not individual members of a set whose members attend to each other.
Research Program
The first stage is purely diagnostic and requires no retraining. We freeze released DINO and D-FINE checkpoints and run COCO train2017 through each one. For every query at every layer we record the hidden state, the box, the class logits and the output-level signals. From these records we fit three kinds of linear probes at each layer. One predicts $r_{q,l}$. One predicts whether $r_{q,l}$ exceeds a small tolerance, meaning the box will still move meaningfully. One predicts whether the query is eventually matched to a ground-truth object. The first two need only the model's own outputs. The third uses the standard Hungarian assignment.
Each hidden-state probe is paired with an observable-feature probe of equal capacity, and with a probe that sees both. The gap between the hidden-state probe and the observable-feature probe measures what the internal state adds. Crossing layer depth with object size, occlusion level and scene density turns the layer-selection disagreement from the language literature into a concrete answer for detectors.
The second stage turns the probe into a gate. At each layer, a query whose predicted remaining refinement falls below a threshold $\tau$ is frozen. Its box and class are emitted as final, and it skips the remaining cross-attention and feed-forward computation. It stays in self-attention as a fixed key and value, so the other queries still see it. This design answers the set-interaction concern raised in the file: frozen queries do not disappear, they simply stop updating. We also test the harsher variant in which frozen queries are removed entirely. Queries confidently predicted to be unmatched background are dropped outright. Sweeping $\tau$ traces an accuracy-versus-compute curve.
The third stage fine-tunes the decoder briefly with the gate active. Surviving layers then learn to work with a partially frozen query set. Measuring this stage tells us how much of the gain is available post hoc and how much requires adaptation.
Baselines
The naive reference is the unmodified detector running all layers for all queries. For D-FINE-L that point is 54.0% AP on COCO at 124 FPS on a T4 GPU, as reported by D-FINE. On tiny-object data, the relevant reference is DINO at 25.9% mAP on AI-TOD-V2, reported by DQ-DETR.
The strongest existing method for adapting compute is DQ-DETR itself, at 30.2% mAP on AI-TOD-V2 and 37.0% on VisDrone. We apply our gate on top of it at its largest query setting. Beating it means matching its mAP with fewer decoder operations, or exceeding it at equal operations.
The strongest competitor for the uncertainty signal is an explicit per-layer head, trained on the same targets and the same data. That head is either a Gaussian variance head with KL loss in the style of He et al., or a small predicted-IoU head. For D-FINE, its free boundary-distribution entropy also serves as a gate. Every gate gets the same threshold sweep and the same fine-tuning budget. Beating these baselines means a curve that sits above theirs, not a single better operating point.
Two further baselines use only visible outputs. One thresholds the class score. The other thresholds the magnitude of the last box update. The ablation that isolates our key ingredient swaps the hidden-state input for the observable features while keeping the target and gate fixed. The second ablation replaces the forward-looking target $r_{q,l}$ with current-IoU error, the backward-looking target that prior work uses.
A separate baseline retrains the detector with fewer decoder layers. It checks whether uniform depth reduction already achieves the same trade-off.
We found no reported early-exit or query-gating result for DETR decoders on Lacuna. The early-exit baselines above are therefore run by us, under identical conditions.
Why not just…?
Why not just stop when the box stops moving? Thresholding the size of the last box update is free and may work well, and this is the most serious threat to the proposal. We do not argue around it. The observable-feature probe includes update magnitude as an input, and the head-to-head comparison measures exactly how much the hidden state adds. Update magnitude cannot foresee that a small step at layer two will be followed by a large correction at layer four, for example when a nearby query claims the object. We will report how often such delayed corrections occur, broken down by crowding level. If the hidden-state gain is negligible, the paper's result is that DETR refinement is effectively monotone and a trivial rule suffices, which is still worth knowing.
Why not just add an explicit variance or IoU head? A small head on the hidden state is architecturally close to a linear probe. The real difference is the target. Existing heads estimate current localization error (He et al.; Uncertainty-aware DETR), not the value of further refinement. They also require retraining the detector. The target-swap ablation tests whether our forward-looking target matters. The frozen-model stage tests whether retraining is needed at all.
Why not just threshold the class score? Detection has a documented mismatch between class confidence and box quality, noted by He et al. and Reconcile Prediction Consistency. A query can be confidently labelled while its box is still poorly placed, and freezing it then locks in the error. We expect class-score gating to lose most at AP75 and AP90, where box precision matters most. The experiments report those strict-threshold metrics precisely to show this.
Why not just train a shallower decoder? D-FINE reports that shallow decoder layers are noisy in compact models, which suggests that uniform depth cuts hurt every query equally. Retrained 3- and 4-layer decoders give a direct comparison. If they match our accuracy-compute curve, per-query gating is unnecessary.
Why not just use multi-pass sampling such as Monte Carlo dropout? Monte Carlo dropout means running the model several times with random units switched off and measuring the spread. It multiplies the very cost we are trying to cut. Both the probe and confidence-network papers cite this multiplied cost as the reason to read internal states instead. We include it only as an upper-bound signal in the diagnostic stage, never as a deployable gate.
Proposed Experiments
The diagnostic stage runs on COCO val2017 with DINO and D-FINE. It reports, per layer, the AUROC for "box will still move" and "query will be matched," and the Spearman correlation for $r_{q,l}$. Each number is given separately for hidden-state, observable and combined probes. Results are stratified by object size and by crowding, using the pairwise overlap annotations of CrowdHuman.
The gating stage reports AP, AP75, AP90 and mMR, the log-average miss rate used on CrowdHuman. Each metric is plotted against decoder FLOPs and measured decoder latency on COCO, CrowdHuman and AI-TOD-V2. We report whole-model latency too, because backbone and encoder may dominate at low query counts. We expect meaningful end-to-end speedups mainly at 900 to 1500 queries.
The first decisive check is whether the hidden-state probe beats the observable-feature probe by a clear margin at the chosen layer. The second is whether, at equal AP to the full model, the gate removes a substantial fraction of decoder computation on CrowdHuman and AI-TOD-V2 while beating update-magnitude gating. The third is whether AP90 holds up, since a gate that quietly freezes slightly misplaced boxes would show up there first.
References
- Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs
- Learning Confidence for Transformer-based Neural Machine Translation
- Bounding Box Regression With Uncertainty for Accurate Object Detection
- Localization Uncertainty-Based Attention For Object Detection
- An Uncertainty-aware DETR Enhancement Framework for Object Detection
- D-FINE: Redefine Regression Task of DETRs as Fine-grained Distribution Refinement
- DQ-DETR: DETR with Dynamic Query for Tiny Object Detection
- DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models
- Reconcile Prediction Consistency for Balanced Object Detection
- CrowdHuman: A Benchmark for Detecting Human in a Crowd