Some search tasks ask a detector to find one specific thing rather than every object: a particular person across thousands of surveillance frames, or an object shown in a single example photo. The first stage of a standard two-stage detector proposes candidate boxes by asking "does this look like an object?" That question is blind to what the user is searching for. Occluded or unusual-looking targets therefore get dropped before any matching happens. Query-guided region proposal networks fix this by fusing the query's features with the scene's features before proposing boxes. The cost is that the expensive fusion must be rerun for every query on every gallery image. Instance Guided Proposal Network for Person Search reports about 6 seconds per query, and that cost grows linearly with the number of queries.
We propose to use knowledge distillation (training a small "student" model to imitate a stronger "teacher") to separate the benefit of query guidance from its cost. The teacher is an early-fusion, query-conditioned proposal network. The student computes a set of proposal "keys" for each gallery image once and caches them. At search time it scores those cached keys against a query vector with a dot product. The student is trained to reproduce the teacher's per-location proposal scores, with extra weight on the distractors the teacher learned to suppress. If this works, query-guided recall comes at close to query-blind cost.
Core Claim
A late-fusion proposal scorer can be trained to approximate an early-fusion teacher's query-conditioned proposal ranking. Trained this way, it keeps most of the teacher's recall at small proposal budgets, especially on hard targets, while making per-query cost negligible. The distilled signal is what matters. The teacher's soft scores over look-alike distractors carry information that ground-truth boxes do not, so the same student architecture trained only on labels should be clearly worse.
This differs from the existing distillation work at this intersection, which all sits after the proposal stage or uses query-blind proposals.
Knowledge Distillation for End-to-End Person Search does two things. It freezes a teacher's identity lookup table and applies it to QEEPS, which reaches 85.0% mAP on CUHK-SYSU. It also shows that distilling from a generic Faster R-CNN teacher harms identity learning. That negative result constrains our design: we never distill generic objectness, only identity-conditioned proposal scores.
Diverse Knowledge Distillation for End-to-End Person Search distills a Re-ID teacher's probabilities and pairwise and triplet relations into the second-stage head, on top of a query-blind detector.
ViLD distills CLIP embeddings into cropped regions, again after class-agnostic proposals.
QSKD selects hard-negative DETR object queries for distillation. Its queries are internal learned slots, not user exemplars. We reuse its main insight: distill on hard negatives rather than easy background.
Method
The teacher $T$ is an IGPN- or QEEPS-style network (Query-Guided End-To-End Person Search). It cross-correlates query and gallery features and outputs, for query $q$ and gallery image $I$, an objectness logit $t_a(q,I)$ at every anchor $a$, plus refined boxes.
The student $S$ has two parts:
- Gallery side. For each image, the gallery encoder emits one key $k_a(I) \in \mathbb{R}^d$ per anchor and a query-independent box regression. These are computed once and cached.
- Query side. The query encoder emits $K$ small vectors $u_1,\dots,u_K \in \mathbb{R}^d$.
The student's proposal score at anchor $a$ is
$$ s_a(q,I) = \max_{j \le K} \; u_j(q)^\top k_a(I) / \tau , $$
where $\tau$ is a learned temperature. Using a max over $K$ query vectors, instead of a single vector, gives the scorer enough flexibility to represent different target parts, such as upper body versus legs under occlusion. We will measure how recall changes with $K$.
The student is trained with the loss
$$ \mathcal{L} = \mathrm{KL}\big(\mathrm{softmax}_{a\in\mathcal{H}}(t_a/\tau_T)\,\|\,\mathrm{softmax}_{a\in\mathcal{H}}(s_a)\big) + \lambda\,\mathcal{L}_{\text{gt}} + \mu\,\mathcal{L}_{\text{box}} , $$
with the following terms:
- $\mathcal{H}$ is the selected anchor set. It contains anchors on the target, plus hard negatives: anchors on other people or objects that the teacher either scored highly or down-weighted relative to a query-blind detector. Easy background anchors are excluded, following QSKD's finding that easy negatives swamp the distillation signal.
- $\tau_T$ is the teacher temperature.
- $\mathcal{L}_{\text{gt}}$ is the usual proposal classification loss on ground truth, weighted by $\lambda$.
- $\mathcal{L}_{\text{box}}$ distills the teacher's refined boxes on its top-ranked anchors, weighted by $\mu$.
At test time the student passes only its top 10 to 20 proposals to the identification head. This follows IGPN's "less is more" finding that fewer, query-conditioned proposals improve matching.
Research Program
We will build teacher and student on a shared backbone and first measure the distillation gap. The gap is the loss in proposal recall at budgets of 5, 10, 20, and 100 proposals when moving from early to late fusion. We will measure it overall and on occluded or crowded targets, as a function of $K$ and $d$.
We then attach an identical second-stage identification head to both the teacher's proposals and the student's proposals. The DKD head is the default. Using the same head isolates the effect of the proposals.
Finally, we test transfer beyond person search. We train the same recipe on the one-shot seen/unseen splits used by CoAE. The question is whether the distilled scorer inherits the teacher's unseen-class guidance, or collapses back to the base-class objectness that Reference Twice identifies as "supervision collapse."
Baselines
Naive reference: query-blind OIM detector. This is the query-blind reference, rerun on our backbone. The file reports 75.5% mAP on CUHK-SYSU for this OIM model (Knowledge Distillation for End-to-End Person Search). The student must beat it clearly on recall at small proposal budgets.
Strongest accuracy competitor: DKD. DKD is a query-blind detector with a relationally distilled Re-ID head. It reports 93.09% mAP on CUHK-SYSU, 50.51% mAP on PRW, and 124 ms runtime (Diverse Knowledge Distillation for End-to-End Person Search). We will reproduce it with our backbone and training schedule. We will then swap only the proposal stage for our student while keeping the DKD head, and tune both under the same budget. Beating DKD means one of two outcomes at comparable total search time over a many-query gallery. The first is higher mAP. The second is equal mAP with a clear gain on the occluded or crowded subset.
Teacher and latency reference: IGPN and QEEPS. IGPN reports 90.3% mAP on CUHK-SYSU, 47.2% on PRW, and about 6 seconds per query. QEEPS reports 88.9% mAP, 98.8% detector recall on CUHK-SYSU, and 37.1% mAP on PRW. We will retrain our own teacher and time teacher and student on the same hardware. The student should recover at least most of the teacher's top-20 recall while cutting per-query cost by an order of magnitude or more.
Ablations isolating distillation. We have three:
- The identical late-fusion student trained with ground truth only.
- The student distilled over all anchors, with no hard-negative selection.
- The student distilled from a generic query-blind detector, which is the $KD_{det}$ setting from Munjal et al.
For the one-shot transfer experiments, CoAE's reported 22.0% AP50 on COCO unseen classes and 63.8% mAP on VOC unseen classes are the reference points.
Why not just…?
Why not just use a high-recall query-blind RPN plus a distilled Re-ID head, as DKD does? This is the most serious threat. DKD's reported 93.09% mAP on CUHK-SYSU already exceeds every query-guided result in the file. IGPN's evidence that hundreds of generic proposals hurt matching was obtained with weaker Re-ID heads. A strong enough head may absorb the distractors. We settle this with the matched-head comparison above. If our proposals give no gain over DKD's proposals under the same head, even on occluded targets and at small budgets, then query guidance is not worth its complexity for person search. We would then report that negative result.
Why not just train the late-fusion student directly on ground truth? Ground truth marks only the target box. It says nothing about which look-alike neighbours are dangerous, and the teacher's soft scores encode exactly that. The ground-truth-only ablation tests this directly. If it matches the distilled student, the distillation claim fails and the contribution reduces to a late-fusion architecture.
Why not just distill a bigger generic detector into the RPN? Knowledge Distillation for End-to-End Person Search found that this pulls features toward generic detection and away from identity. Our distillation targets identity-conditioned scores, and the $KD_{det}$ ablation checks that the difference is real.
Why not just shrink the teacher's fusion module? IGPN's improved cross-correlation layers already cut RPN parameters from 254M to 10M. The fusion still runs once per query-image pair, so the linear scaling in the number of queries remains. Caching gallery-side keys removes that scaling. Pruning does not.
Decisive Checks
The idea stands or falls on four results on CUHK-SYSU and PRW:
- The distilled student recovers most of the teacher's top-20 recall on occluded targets.
- It beats the ground-truth-only student with the same architecture.
- Under the DKD head, it improves on DKD's query-blind proposals at matched total search cost.
- Its per-query time is negligible compared with the teacher's.
A secondary check is the COCO/VOC unseen-class split, measured against CoAE. It tests whether the distilled guidance survives on classes the student never saw during training.
References
- Instance Guided Proposal Network for Person Search
- Knowledge Distillation for End-to-End Person Search
- Diverse Knowledge Distillation for End-to-End Person Search
- Open-vocabulary Object Detection via Vision and Language Knowledge Distillation
- Knowledge Distillation via Query Selection for Detection Transformer
- Query-Guided End-To-End Person Search
- One-Shot Object Detection with Co-Attention and Co-Excitation
- Reference Twice: A Simple and Unified Baseline for Few-Shot Instance Segmentation