Learning to Merge Tiles in Simulation: Seam Handling as a Transferable Skill for Aerial Detection

Drone and satellite images are too large to fit in GPU memory, so they are cut into a grid of tiles and each tile is processed on its own. Any object lying across a tile edge gets split. It is then detected twice, detected as two fragments, or missed. Today this is patched after the fact: tiles overlap, and duplicate boxes are removed with a hand-tuned suppression rule. We propose to replace that rule with a small learned module. For every detection it predicts three things: whether the object is cut off by the tile edge, what the object's full extent is, and which tile "owns" it. The module is trained entirely in a simulator, which knows every object's full extent even where a tile boundary hides part of it.

Our central bet is that seam handling is mostly a geometric skill rather than an appearance skill, so it should cross the synthetic-to-real gap far better than a detector's appearance features do. Deciding that half a car at the tile's right edge continues into the next tile depends on edge position, box shape, and features along the border band. It does not depend on how realistic the simulator's paint and lighting look. If the bet holds, real-world tiled detection can recover seam-induced errors without real labels, even where the detector's recognition accuracy stays limited by the domain gap. If it fails, we learn that seams are entangled with appearance, which is also worth knowing.

Why this gap is open

Two strands of work frame the problem. The tiling literature treats boundaries purely at the output level. The Power of Tiling for Small Object Detection uses 25%-overlapping grids and a modified non-maximum suppression, the rule that deletes lower-scoring boxes overlapping a higher-scoring one. With this setup it raises VisDrone2018 mean average precision (mAP) from 11% to 36%, at the cost of up to 15 forward passes per image. A Large Contextual Dataset for Classification, Detection and Counting of Cars with Deep Learning averages car counts over offset strides and explicitly leaves "more elegant boundary handling" as future work. Boundary Loss for Remote Sensing Imagery Semantic Segmentation sharpens contours near tile edges but only for binary masks, and does nothing about object identity across tiles. Together these papers motivate the proposal: they document seam errors and fix them heuristically.

The multi-scale strand fuses a low-resolution global view with high-resolution local features. Examples are MGDFIS, which reports a Faster R-CNN AP50 of 53.2% on VisDrone 2019, and the dilated streams of UAVid. Both assume the whole image is resized to network size, so tiles never appear. We reuse their global-thumbnail idea as one input to our module. We do not make it the contribution.

On the synthetic side, SKYSCENES reports aerial sim-to-real drops of up to 50%. It reaches 47.09 mIoU on real UAVid with DAFormer, but evaluates full or natively cropped frames and never analyzes tiling. RetinaGAN narrows the gap by translating appearance under a detector-consistency loss. That targets appearance, which is the part of the problem we argue matters least for seams.

Object-centric dataset bias enters as the explanation for why seams hurt so much. Backbones pretrained on iconic, centered-object images, or by contrastive cropping that assumes one dominant object per image (as analyzed by Zhao et al. and Object-aware Cropping), have rarely seen objects sliced by the frame edge. A tile edge is therefore out of distribution for them. Multiple Object Stitching (MOS) is the closest existing work touching seams. It stitches whole single-object images into grids to pretrain multi-object representations. Its seams fall between objects and never through them, and it does no high-resolution inference, so it addresses a different seam from ours.

Core Claim

A cross-tile merging module can be trained only on simulated aerial scenes, with exact full-object extents as free supervision. Attached to a detector running on real tiled imagery, it removes most of the extra error concentrated at tile boundaries: duplicates, fragments, and missed objects. It matches or beats 25%-overlap tiling with suppression at lower compute. We also expect it to come close to the same module trained on real labeled tiles.

The ownership rule is the key simplification. Let $b$ be a detection's predicted full box, $c(b)$ its center, and $R_t$ the core region of tile $t$, meaning the tile minus the half-overlap margin shared with its neighbors. Tile $t$ keeps the detection only if

$$ c(b) \in R_t . $$

The core regions partition the image, so each object is kept exactly once, and no suppression threshold needs tuning. The module's job is to predict $b$ well enough for cut-off objects that $c(b)$ lands on the correct side of the edge.

Research Program

The module sits on top of any tile-level detector. Its inputs are the tile's own features, a narrow band of border tokens borrowed from the four neighboring tiles, and a coarse global thumbnail of the full image. For each detection it predicts three quantities: a truncation flag, a full-extent box, and ownership. It does this through cross-attention restricted to the border band, which keeps extra compute small and roughly constant per tile.

Supervision comes from SkyScenes. We render or crop scenes at the native resolution and tile them under randomized grid offsets and tile sizes. For every visible fragment we record the full instance extent from the simulator's instance labels. The simulator's controls over altitude (15, 35, 60 m) and pitch, together with its pedestrian generator, let us vary how many small objects straddle seams and train across that range.

Training happens in two stages. First we train the detector itself on SkyScenes, with or without real fine-tuning depending on the regime. Then we freeze it and train the merging module on simulated tiles. Freezing matters for the science: it lets us swap the source of the merger's training data while holding recognition fixed, which is how we separate the seam effect from the appearance effect.

To measure the phenomenon we introduce a seam-band evaluation. Every ground-truth object in a real image is labeled interior or seam-crossing relative to the tile grid used at inference, and we report AP separately for each group. We also report a duplicate rate and a fragment rate. The quantity of interest is the seam excess error: interior AP minus seam-band AP. Appearance mismatch between simulation and reality should lower both groups. Only seam handling should widen the difference between them.

Finally, a diagnostic study tests the object-centric-bias explanation. We measure seam excess for the same detector head on backbones pretrained three ways: supervised ImageNet, DINO-style contrastive learning, and MOS. The prediction is that iconic-pretrained backbones show the largest seam excess before merging, and that the learned module shrinks the differences between them.

Baselines

The naive reference point is non-overlapping tiling with plain per-tile outputs concatenated. This is the floor, and it defines the seam excess we aim to remove.

The strongest existing seam method is overlap tiling with modified suppression, following The Power of Tiling. That paper reports 36% mAP on VisDrone2018 with 5x3 tiling for its Pelee38 detector. We re-run this approach on our shared detector at 25% overlap and tune the suppression threshold on the validation split, with the same tuning budget we give our module. We compare two ways: at matched tile grids, and at matched total compute. The overlap method pays for every extra pass. Beating it means higher seam-band AP and a lower duplicate rate at equal or lower compute, with interior AP no worse.

For context on the fully supervised real regime, VisDrone-DET2019 reports 29.62% AP for the top ensemble, and MGDFIS reports 53.2% AP50. These are recognition-quality references, not seam baselines. No reported result found on Lacuna for seam-band AP or duplicate rate on any aerial benchmark, so we establish these ourselves.

For segmentation transfer, the reference is SkyScenes-trained DAFormer at 47.09 mIoU on UAVid. We tile UAVid at native resolution and report whether merging-aware inference changes human-class IoU, which UAVid reports below 5% under downsampling.

Three ablations isolate the key ingredient. The first gives the module only the global thumbnail, with no border tokens and no full-extent head; this tests whether a generic multi-scale context branch already solves the problem. The second removes ownership and uses the full-extent boxes with standard suppression instead. The third trains the same module on real VisDrone tiles, whose full-image boxes also provide extents beyond each tile. This real-trained module serves as an upper bound. The ratio of sim-trained to real-trained seam-excess reduction is the headline transfer number.

Why not just…?

Why not just overlap tiles and suppress duplicates? It works, as its 11%→36% jump shows, and it is our main competitor. It has two structural costs. Compute grows with overlap. And a fixed threshold must trade merging adjacent true objects, common in dense pedestrian scenes, against leaving duplicates in place. The matched-compute comparison and the duplicate and fragment rates in dense VisDrone scenes will show whether the learned module actually improves on it or only matches it more cheaply. Matching it at lower compute would still be a useful result.

Why not just train the merger on real labeled tiles? Where dense real box labels exist, this is a real threat. Tiling a labeled VisDrone image gives full extents for free, so simulation adds nothing for boxes there. We therefore do not claim simulation beats real labels. Our claim is conditional and targets deployments without such labels, such as a new sensor or region, instance masks rather than boxes, or occluded objects whose true extent the simulator knows but annotators do not. The real-trained merger is included as an upper bound, and the sim/real ratio decides the claim. A ratio well below one would falsify the "geometric, hence transferable" hypothesis.

Why not just use larger tiles or downsample the whole image? Downsampling destroys the small objects that motivate tiling; UAVid humans fall below 5% IoU under this regime. Larger tiles reduce the number of seams but not the per-seam error, and they hit memory limits. Our tile-size sweep reports seam excess as a function of tile size to show this directly.

Why not just close the appearance gap with image translation or sim pretraining followed by real fine-tuning? Approaches like RetinaGAN change textures, not tile geometry. They need an already-competent detector as critic, and they should raise interior and seam-band AP equally, leaving seam excess unchanged. We test this by applying our module on top of a sim-pretrained, real-fine-tuned detector. If seam excess persists there and our module removes it, the two fixes are complementary rather than competing.

Decisive Checks

The proposal stands or falls on three measurements. First, real VisDrone and UAVid must show a nonzero seam excess under naive tiling. If tile boundaries cost little, the premise collapses. Second, the sim-trained module must remove a substantial share of that excess while leaving interior AP unchanged. That is the signature of a geometric fix rather than an appearance fix. Third, its reduction must approach the real-trained upper bound and must clearly exceed the thumbnail-only ablation. Otherwise the gain is ordinary multi-scale context, not learned seam ownership.

Car counting on COWC scenes offers a secondary check. Its prior fix is stride averaging, so counts from the ownership rule should agree with stride-averaged counts in a single pass. The pretraining study then either supports object-centric bias as the reason seams hurt, or cleanly rules it out.