Carrying Object Slots Across Image Tiles

Large images such as aerial orthomosaics, drone frames and tissue slides are too big for a vision transformer to process in one pass, because the cost of self-attention (every image patch comparing itself to every other patch) grows with the square of the number of patches. Practitioners therefore cut the image into tiles and process each tile separately. Objects that straddle a tile cut get split in two: a tree crown becomes two half-crowns, a building becomes two fragments, and a car is counted twice. Object-centric models are a natural fit for these scenes. These models summarize an image as a small set of "slots," one learned vector per object, found by letting slots compete for image patches in a step called Slot Attention. The trouble is that they assume the whole scene fits in one forward pass, and they use a fixed number of slots chosen before training.

We propose to process tiles in a fixed scan order and carry slots from one tile into its neighbors. When a carried-in slot meets new tiles, it either keeps explaining the rest of its object or retires, and fresh slots are opened only for objects that have not been seen yet. The training signal that makes this learnable is a boundary-weighted reconstruction loss. Each slot leaving a tile must predict the frozen backbone features in a band just across the seam, with a weight that decays with distance from the cut. The model is therefore graded on continuing objects across tile edges and not only on explaining the inside of each tile. Cost stays linear in the number of tiles, because each tile only runs attention between its own patches and a small number of slots. The total number of slots across the scene grows with the scene instead of being fixed in advance.

Core Claim

Object bisection at tile seams is an identity problem, not a pixel-smoothing problem. Pixel-level seam fixes cannot solve it. That includes the center-weighted sliding-window blending of Alleviating tiling effect by random walk sliding window in high-resolution histological whole slide image synthesis, which averages overlapping predictions, and the neighbor-pixel padding of Efficient High-Resolution Image Editing with Hallucination-Aware Loss and Adaptive Tiling, which removes seams without overlap. Both make the image look continuous but never decide whether two half-objects are one entity. Slots are discrete, countable carriers of identity. We claim that a slot carried across a seam and trained to predict what lies just beyond it binds bisected objects more reliably than the strongest simple alternative, which runs slot models per tile and matches slots afterwards. We also claim it does this at 0% tile overlap, so the matching baseline's extra compute for overlap is not needed.

How the pieces fit, and what prior work supplies

Most of the method is borrowed. From Bridging the Gap to Real-World Object-Centric Learning (DINOSAUR) we take slots that reconstruct frozen self-supervised ViT features rather than pixels; DINOSAUR showed this is what makes slot models work on COCO-scale real images. Its finding that simple MLP decoders force slots to specialize into instances also constrains our decoder choice. We avoid the diffusion decoder of Object-Centric Slot Diffusion both because of its training cost and because its authors report that an overly expressive decoder can ignore slots. From Adaptive Slot Attention: Object Discovery with Dynamic Slot Number (AdaSlot) we take a Gumbel-softmax gate that switches candidate slots on or off, plus its capacity regularizer. AdaSlot gates slots within one fixed-size image. We instead gate only the fresh slots in each tile, while carried-in slots retire when their attention mass in the new tile falls to zero. Scene-level capacity becomes a running count rather than a hyperparameter. Video object-centric models such as the Dynamic Temporal Slot Transformers of Guiqiu Liao et al. already add and drop slots over time. Our setting differs in two ways. Neighbors are spatial and two-sided, so a tile inherits slots from both its left and top neighbors, and two inherited slots may turn out to be the same object and need merging. And there is no motion cue to help with binding.

The seam loss follows the distance-decay weighting used in outpainting, such as the spatial discounting of Yu et al. and the cosine decay masks of Very Long Natural Scenery Image Prediction by Outpainting. Those works down-weight far-from-boundary pixels because predictions there are legitimately uncertain. Here that same uncertainty matters: a slot can only be expected to predict a short way past the cut. Concretely, let $s_k$ be a slot that has nonzero attention within distance $b$ of the shared edge of tiles $A$ and $B$. Let $f_j$ be the frozen backbone feature at patch $j$ of tile $B$, $d_j$ its distance in patches from the edge, $m_{kj}$ the decoder's mask for slot $k$ at $j$, and $\hat f_{kj}$ its predicted feature. The seam term is

$$ \mathcal{L}_{seam}=\sum_{j:\,d_j\le b} \gamma^{d_j}\,\Big\| f_j-\sum_k m_{kj}\hat f_{kj}\Big\|_2^2, $$

with decay $\gamma\in(0,1)$. It is added to DINOSAUR's uniform within-tile feature loss and AdaSlot's capacity term. This is a new use of boundary weighting. Earlier uses either smooth generated pixels at an inset boundary, as in InsetGAN for Full-Body Image Generation, Conditional Infilling GANs for Data Augmentation in Mammogram Classification and Image Inpainting via Generative Multi-column Convolutional Neural Networks, or sharpen segmentation contours, as in Boundary Loss for Remote Sensing Imagery Semantic Segmentation. Here the weighting shapes which slot claims which patch across a cut. The edge-feature decoding in mFRESA, from Evaluating Object-Centric Models beyond Object Discovery, is the closest object-centric precedent. It applies uniform weight to edges inside an image and does not consider tile interfaces.

On efficiency, each tile with $n$ patches costs one backbone pass plus slot attention of order $(K_{in}+K_{new})\cdot n$. Here $K_{in}$ is the number of carried-in slots and $K_{new}$ the number of fresh candidates. A scene of $T$ tiles therefore costs $O(T\cdot n\cdot K)$ instead of $O((Tn)^2)$ for global attention. Compression schemes like LookupViT: Compressing Visual Information to a Limited Number of Tokens achieve similar savings with a fixed set of compressed tokens. Their authors note that these tokens have not been validated for dense localization, and they are not countable objects. Our carried slots play the same role as a small memory crossing a large canvas, but each one is meant to be an object.

Research Program

We build three variants of the system on one frozen DINOv2 backbone with an MLP decoder. The first carries slots across tiles but uses only uniform reconstruction loss. The second adds the seam loss with distance decay. The third adds a merge step: when slots inherited from the left and top neighbors both claim the same patches near the shared corner, attention competition decides, and the losing slot retires. We analyze how the decay $\gamma$ and band width $b$ trade seam binding against interior over-grouping, the boundary-loss analogue of the hallucination-versus-consistency tension noted in the inpainting literature. We also measure whether scan order matters by comparing a raster scan with a serpentine scan and with averaging over the four corner-start orders.

Baselines

The naive reference point is per-tile AdaSlot with no cross-tile reasoning, run on the same 0%-overlap tiles with masks simply stitched together. It shows how much bisection costs. A second reference point is whole-image DINOSAUR or AdaSlot after downsampling the full scene to 224 or 518 pixels. It tests whether tiling is needed at all. In The Power of Tiling for Small Object Detection, downsampling 4K drone frames dropped mAP to 11%, compared with 36% for 5×3 tiling. We expect a similar collapse for small crowns and buildings, but we will measure it.

The strongest competitor is per-tile AdaSlot on tiles with 25% overlap, followed by post-hoc slot matching. Slots in adjacent tiles are paired by Hungarian matching on a cost that combines slot-feature cosine distance and mask IoU within the overlap. Matched slots are fused, and fused masks are blended with the center-weighted map of the random-walk sliding-window method. This is the tiling community's overlap-and-merge practice moved to slots, and a reviewer would reasonably expect it to work. For fairness it uses the same backbone, the same per-tile resolution, the same training data and the same hyperparameter search budget, including a search over matching thresholds. We report it both at its natural overlap and at a reduced tile count that matches our FLOPs. A context-padding baseline gives each 0%-overlap tile a margin of neighbor patches in the style of MobilePicasso's adjacent padding, then applies the same matching. This isolates whether extra context alone fixes binding.

For the strongest numbers available, no reported result was found on Lacuna for any slot model on tiled high-resolution imagery. The reported references are on untiled images: AdaSlot reaches ARI 39.00 on MS COCO against 20.83 for a fixed 33-slot model, and DINOSAUR reaches 67.2 CorLoc on COCO-20k. Beating the strongest competitor means one of two outcomes at matched FLOPs. Either we get higher seam-straddling binding accuracy with whole-image FG-ARI and mBO no worse than the baseline's, or we get equal binding at substantially lower compute because no overlap is needed. The key ablation is carried slots with uniform loss against carried slots with the seam loss. Carrying alone might already bind objects, and only this comparison tells us whether the boundary weighting earns its place. A second ablation replaces the $\gamma^{d}$ decay with flat seam weights.

Why not just…?

Why not just tile with overlap and merge slots afterwards? This is the real threat, and the program is designed around it. Matching can work when both halves of an object are large enough to produce confident slots with similar features. It should fail when one fragment is a sliver, because per-tile slot models tend to absorb small fragments into background or neighbor slots. In that case there is nothing on the other side to match. To test this directly, we stratify binding accuracy by the fraction of each ground-truth object on its smaller side of the cut. If the matching baseline equals our method across all strata at matched FLOPs, we will report that carrying slots is unnecessary, and the contribution reduces to a benchmark and a negative result.

Why not just set a large fixed number of slots per tile? AdaSlot's own COCO result answers this. A fixed 33-slot model scored ARI 20.83 against 39.00 for gated slots, and Marissa A. Weis et al. describe surplus slots fragmenting objects. Across tiles the problem gets worse, because a fragmented object yields several slot pieces to reconcile. We include the large fixed-$K$ variant as a run anyway.

Why not just use a uniform loss and let carried slots learn continuity on their own? It may suffice, which is why it is our central ablation. The argument against it is that a uniform within-tile loss gives a slot no reward for explaining patches it has not yet seen. Only the seam term penalizes a slot that retires early.

Proposed Experiments

The controlled testbed is tiled COCO. COCO images are upsampled, cut into 2×2 and 4×4 grids processed at 224 pixels per tile, and evaluated against COCO instance masks. Because cut positions are known, every straddling object is identified. We report whole-image FG-ARI and mBO, plus seam binding accuracy: the fraction of straddling ground-truth objects whose stitched mask is dominated by a single scene-level slot. We also report the double-count rate and the error between predicted and true scene-level object counts. Zero-shot transfer to PASCAL VOC follows DINOSAUR's protocol.

The real-scale tests use SelvaBox orthomosaics (SelvaBox: A high‑resolution dataset for tropical tree crown detection, resource SelvaBox) and ISPRS Potsdam and INRIA building tiles. On SelvaBox we convert slot masks to boxes and score with the dataset's raster-level $RF1_{75}$, which evaluates at full-raster scale and so penalizes double-counted crowns. The supervised DINO Swin-L detector's reported 48.60 is a ceiling for context, not a target for an unsupervised model. The comparison that counts is against the slot baselines. For Potsdam and INRIA, no reported object-discovery number was found on Lacuna; we use building instances derived from connected components of the labels. All methods report FLOPs and peak memory per scene.

The decisive check is the stratified binding analysis on tiled COCO and SelvaBox. The method is justified only if it beats overlap-and-merge on small-fragment straddlers at matched compute without losing interior FG-ARI, and if removing the seam loss removes most of that gain.