When a detector must learn a new object class from one to thirty labeled examples, it usually sees those few examples at only one size. The best-known fix generates many resized copies of each labeled object, so the detector learns to find it at every size. That fix crops the object out of its scene first. Work on tiny targets says this is backwards for small sizes: a 10-pixel car has no visible internal structure, and the detector recognizes it mostly from what surrounds it, such as the road it sits on. We propose to rebuild the multi-scale copies with the surroundings kept in. Each labeled object is cut out with a window of its real neighborhood, and the window grows as the object is rendered smaller. Each rendered copy is then fed to the detector as a guaranteed positive example at the matching pyramid level.
The setting where this matters most is few-shot detection in aerial and remote-sensing imagery. There, most instances are small, scale varies over orders of magnitude, and annotation is expensive. The test is simple and falsifiable. If context-preserving pyramids help, the gain should show up on small novel-class objects in aerial benchmarks, it should shrink on natural-photo benchmarks where small objects are rarer, and it should disappear when the preserved context is swapped for context taken from an unrelated image.
Core Claim
Multi-scale Positive Sample Refinement for Few-Shot Object Detection (MPSR) names the central problem as scale sparsity: novel classes appear at too few sizes. It also identifies why naive multi-scale augmentation fails with few examples. Boxes that partially overlap a resized object get labeled as background ("improper negatives"), and in a data-starved regime these wrong labels swamp the scarce positive signal. MPSR's answer has three parts. It crops each ground-truth object and resizes it from $32^2$ to $800^2$ pixels. It routes these crops through an auxiliary branch that shares weights with the main detector and is dropped at inference. It assigns each crop directly as a positive to its pyramid level. MPSR supplies our training machinery.
Finding Tiny Faces supplies the constraint MPSR ignores. It shows that features for tiny and large instances of the same class are qualitatively different, and that good tiny-object templates draw most of their receptive field from outside the box. Context-Transformer: Tackling Object Confusion for Few-Shot Detection motivates the claim from the few-shot side. It finds that scene context is what resolves classification confusion when novel examples are scarce, reaching 43.8 versus 39.4 mAP on 5-shot VOC with a 15.6K-parameter attention module on SSD.
Our claim is that the information MPSR discards, the real neighborhood of each labeled instance, is exactly what a few-shot detector needs most at small rendered scales. For a labeled box of side $b$ rendered at target side $s$, we crop a square window of side $c(s)\,b$ around it and resize the window so the object lands at side $s$. Here $c(s)$ is a context multiplier that decreases with $s$:
$$c(s) = \max\!\left(1,\; c_{\max}\left(\tfrac{s_0}{s}\right)^{\gamma}\right),$$
where $s_0$ is the smallest rendered size, $c_{\max}$ is the context multiplier used at that smallest size, and $\gamma \ge 0$ controls how fast context shrinks as the rendering grows. Setting $\gamma=0$ and $c_{\max}=1$ recovers MPSR exactly, which makes the key ingredient directly ablatable. The decreasing shape follows the Tiny Faces finding that the context share of a useful template rises as the target shrinks.
Research Program
We build on the two-stage transfer recipe of Frustratingly Simple Few-Shot Object Detection (TFA). That recipe pretrains a Faster R-CNN with a feature pyramid on base classes, then fine-tunes on a small balanced support set. TFA froze everything except a cosine-normalized box head and still beat meta-learning by 2 to 20 AP. On top of this we add an MPSR-style auxiliary branch that receives context-preserving renderings instead of isolated crops.
Keeping context creates two problems that MPSR never faced, and solving them is most of the engineering. First, a context window can contain other objects. Some are annotated base or novel instances, which we re-project and label normally. Others are unannotated instances of the novel class, since a K-shot split annotates only K instances. We cannot safely call these background, so any region the pretrained detector's class-agnostic proposals score highly but no label covers is marked "ignore" rather than negative. This extends MPSR's improper-negative logic from partial overlaps to unlabeled neighbors. Second, windows near image borders are padded with a learned constant, and we record the padded fraction so that we can check whether gains come from real pixels.
The study has three parts. The first is the method, with $c_{\max}$ and $\gamma$ chosen on a held-out base-class validation split rather than on novel classes. The second is a mechanism analysis that breaks AP down by novel-object size. The third is a context-content control. In it, each preserved window is replaced by a window of the same size taken from a different training image, with the object pasted in at the same scale. This separates "more pixels around the object" from "the right pixels around the object." Training cost matches MPSR plus the larger crops, and inference cost is unchanged because the auxiliary branch is discarded.
Baselines
The naive reference point is a base-trained Faster R-CNN with a feature pyramid, fine-tuned on the support set with standard single-scale training. Alongside it we run TFA's frozen-backbone cosine head, which is the strongest simple method and the default any reviewer will ask about. TFA reports gains of 2 to 20 AP over meta-learning on VOC and COCO but no single headline number in our file. We rerun it on every benchmark.
The strongest existing method on the exact question is MPSR, whose reported results are 41.7% novel mAP on PASCAL VOC 1-shot and 14.1% on MS COCO 30-shot (Multi-scale Positive Sample Refinement for Few-Shot Object Detection). It shares our detector, branch design and assignment rule. It differs only in discarding context, so it doubles as the $\gamma=0, c_{\max}=1$ ablation.
For aerial few-shot detection, the file's concrete reported numbers come from Analyzing the Impact of Low-Rank Adaptation for Cross-Domain Few-Shot Object Detection in Aerial Images. That paper fully fine-tunes DiffusionDet, a detector that refines boxes by iterative denoising, and then trains low-rank adapters on the few shots, reporting 11.64 mAP on DIOR 1-shot and 4.97 mAP on DOTA 1-shot. Its detector and protocol differ from ours, so we rerun its recipe under our splits and do not compare against its printed numbers. Transformation-Invariant Network for Few-Shot Object Detection in Remote Sensing Images (TINet) adds a pyramid to a meta-learning detector and enforces consistency under flips. It is the most domain-matched baseline on DIOR and NWPU VHR-10.v2, but no reported per-shot result was found on Lacuna for TINet, so it is reimplemented.
Context-Transformer stacked on MPSR tests whether adding context through feature attention matches adding it through training data. We port its affinity module to the pyramid detector.
Fairness rules apply to every method. All share the same base-pretrained backbone, the same support sets per seed, the same fine-tuning iteration budget, and an equal hyperparameter search of 12 configurations, selected on base-class validation. Seeds follow TFA's stability finding: 30 runs on VOC, 10 on COCO, and 10 on each aerial benchmark. Beating MPSR means a novel-class AP gain whose 95% interval over seeds excludes zero, with the gain concentrated in small objects rather than spread uniformly.
Why not just…?
Why not just do multi-scale jitter of whole images, with partial overlaps marked "ignore"? This is the most serious threat. Rescaling the full image keeps context for free, and ignore labels address improper negatives without MPSR's machinery. The limitation is range. A 12-pixel vehicle in a 1024-pixel aerial tile can only be enlarged to mid-pyramid sizes by blowing up the whole tile to impractical memory. Shrinking a large object shrinks its context with it, so the context share never changes with scale. Our method decouples the two. We run this baseline at matched GPU memory, and if it equals our method on novel small-object AP, we will conclude that scale-dependent context is unnecessary and report that.
Why not just add Context-Transformer on top of MPSR? Context-Transformer injects context at inference through attention weighted by the source detector's scores. That same file notes this weighting becomes noisy when backgrounds are unannotated or visually indistinct, as in nadir aerial views. It also leaves MPSR's training signal context-free, so the pyramid features learned for small renderings never see surroundings. The stacked baseline above settles whether the two routes to context are interchangeable.
Why not just unfreeze the pyramid with low-rank adapters, letting it adapt to novel scales? The LoRA aerial study shows adapters give modest gains at 1 shot and saturate by 10 to 50 shots, where full fine-tuning catches up. Adapters change what parameters move, not what scale and context the training signal covers, so the two are complementary rather than competing. We include TFA plus pyramid-level adapters and the combination of adapters with our method. If adapters alone close the small-object gap, the data-side argument weakens.
Why not just run test-time image pyramids? Multi-Scale Cross Distillation for Object Detection in Aerial Images reports that its single-pass model runs about seven times faster than multi-scale testing. Tiny Faces also shows that upsampling at test time cannot compensate for templates that never learned context-dependent small-object features. We report test-time pyramids as an accuracy ceiling at its real latency, not as a method to beat at equal cost.
Proposed Experiments
The primary benchmarks are DIOR and NWPU VHR-10.v2 under the base/novel splits used by TINet at 5, 10, 20 and 30 shots, plus 1-shot DIOR for comparison with the LoRA study. We use horizontal boxes so that all baselines run unchanged. PASCAL VOC (1 to 10 shots, three splits) and COCO (10 and 30 shots) serve as the natural-photo control where the effect should be smaller. DOTA 1-shot is a stress test, given that more than 57% of its instances are 10 to 50 pixels.
Metrics are novel AP and base AP, novel AP broken down by object size (COCO small/medium/large thresholds), and the false-positive rate on background clutter. The clutter rate checks the "context contamination" failure, where extra context injects background noise.
Three checks decide the outcome. First, against MPSR, the novel small-object AP gain should exceed the large-object gain on DIOR and DOTA. Second, the shuffled-context control should erase most of the gain. If it does not, the benefit comes from extra pixels or regularization, not from context. Third, sweeping $\gamma$ should show that a decreasing context multiplier beats any constant multiplier at matched average crop size. A flat or reversed result would falsify the Tiny Faces-motivated design and point toward the simpler whole-image jitter baseline.
References
- Multi-scale Positive Sample Refinement for Few-Shot Object Detection
- Finding Tiny Faces
- Context-Transformer: Tackling Object Confusion for Few-Shot Detection
- Frustratingly Simple Few-Shot Object Detection
- Analyzing the Impact of Low-Rank Adaptation for Cross-Domain Few-Shot Object Detection in Aerial Images
- Transformation-Invariant Network for Few-Shot Object Detection in Remote-Sensing Images
- Multi-Scale Cross Distillation for Object Detection in Aerial Images