Pheromone Memory for Deciding Where a Video Detector Looks Closely

Small, distant objects are the hardest part of detection on robots, drones and cars. A pedestrian 150 meters ahead may cover a few dozen pixels. The expensive fix is to upsample or re-detect the whole frame at high resolution. The cheaper fix, shown by Patch-based Selection and Refinement for Early Object Detection (DPR), is to split the image into an 8×8 grid and send only the patches that seem to contain objects to a costly refinement module. That paper reports that only about 22.8% of patches contain objects and that compute drops by 77.2%. Its weakness, which the authors state themselves, is that a patch the selector skips can never be recovered. In a video this happens on every frame, and each frame's decision ignores everything learned from earlier frames.

We propose to replace the per-frame selector with a pheromone map. This is a grid of numbers, one per patch, that persists across frames. It rises where heavy refinement found objects and fades everywhere else. At each frame, a fixed budget of $k$ "ants" samples patches for refinement. Each ant combines the pheromone value with a cheap per-frame cue. Borrowing from the bounded Max-Min Ant System (Simple Max-Min Ant Systems and the Optimization of Linear Pseudo-Boolean Functions), pheromone is clamped between a floor and a ceiling. The floor guarantees that every patch is revisited within a known expected number of frames, however confident the system becomes. In short, a detector's memory of where objects have recently been should decide where it spends its expensive computation, and a lower bound on that memory turns "the selector may miss things forever" into "the selector misses things for at most a known number of frames."

Core Claim

The claim is narrow. For streaming detection of small objects under a fixed per-frame refinement budget, a scheduler with persistent, evaporating memory and a bounded exploration floor gives better tiny-object accuracy and faster first detection of newly appearing objects than per-frame patch selection at matched compute. The scheduler reaches this by trading a small amount of exploration for a guaranteed bound on how long any missed object stays missed.

Ant colony optimization supplies the mechanism. The modular-design direction supplies the decomposition that makes it testable: the cheap detector, the heavy refiner, and the scheduler sit behind fixed interfaces, so each can be swapped independently. The accuracy-efficiency trade-off is the regime in which everything is measured.

We are deliberately not proposing the obvious intersection, ant-colony search over detector architectures. The reasons are in the "Why not just…?" section. That question is already served by stronger search methods, and it would not use what is distinctive about ant colonies: memory that lives in a shared environment and keeps adapting online.

Mechanism

Let patch $i$ in the grid carry pheromone $\tau_i$ and a cheap heuristic $\eta_i$. The heuristic could be the maximum class confidence from a lightweight detector run on the downsampled frame, or a simple frame-difference energy. The $k$ ants choose patches without replacement with probability

$$p_i = \frac{\tau_i^{\alpha}\,\eta_i^{\beta}}{\sum_j \tau_j^{\alpha}\,\eta_j^{\beta}},$$

where $\alpha$ and $\beta$ weight memory against the current frame. After refinement, each patch is updated as

$$\tau_i \leftarrow \mathrm{clip}\big((1-\rho)\,\tau_i + \Delta_i,\ \tau_{\min},\ \tau_{\max}\big).$$

Here $\rho$ is the evaporation rate. $\Delta_i$ is a deposit equal to the refined detection confidence in patch $i$, and zero if the patch was not refined. We will also test the exponential deposit rule of A Deterministic Model for Analyzing the Dynamics of Ant System Algorithm and Performance Amelioration through a New Pheromone Deposition Approach and Extension of Max-Min Ant System with Exponential Pheromone Deposition Rule. In our setting that rule means deposits grow sharply with confidence, so a few strong detections dominate the memory. Deposits may also diffuse to the eight neighboring patches, because moving objects cross patch boundaries. This is the same neighborhood logic that lets ants trace coherent contours in Development of An Ant Colony Optimisation-Based Edge Detection Framework.

The floor $\tau_{\min}$ gives every patch a minimum per-frame selection probability $q>0$ that depends only on $\tau_{\min}$, $\tau_{\max}$, the range of $\eta$, and $k$. A missed object therefore goes unrefined for at most about $1/q$ frames in expectation. This follows from the geometric waiting time of independent per-frame trials. The prediction is testable: the distribution of "frames until first detection" should have a tail cut off near $1/q$.

The known weakness of ant-colony methods is sensitivity to $\rho$, $\alpha$, $\beta$ and greediness. DeepSwarm (DeepSwarm: Optimising Convolutional Neural Networks using Swarm Intelligence) found that extreme greediness stalls and low greediness becomes a random walk. This constrains the design, so we will report full sensitivity sweeps rather than a single tuned point. The per-frame cost of the scheduler is negligible: 64 numbers and one sampling step. That sidesteps the wall-clock cost that makes pixel-level ant walks impractical in the edge-detection work.

Research Program

We will build three swappable components behind fixed interfaces. The first is a cheap pass: a small detector on the downsampled frame that emits $\eta$ and base detections. The second is a refiner: either DPR's conditional-diffusion super-resolution followed by re-detection, or simply the cheap detector run on a native-resolution crop. The third is a scheduler: the pheromone scheduler, or any baseline selector. We will build on the mmdetection-based PointTinyBenchmark, which already contains tiny-object heads and scale-matching code.

The interface design follows the decoupling argued for in FaceX-Zoo: A PyTorch Toolbox for Face Recognition and DeepDIVA: A Highly-Functional Python Framework for Reproducible Experiments. Those toolboxes assume a static forward pass. Our scheduler interface must also carry state across frames and choose a variable set of patches, which is exactly where the file notes that static modular APIs fall short. Making this stateful interface explicit is a secondary, engineering contribution. The primary purpose of the modular structure is scientific: it lets us attribute gains to the scheduler alone, by holding the cheap pass and the refiner fixed while swapping schedulers.

We will also run a second, non-temporal setting: tiling very large aerial rasters. Here the ants move spatially across tiles instead of across time, and pheromone spreads to neighboring tiles, because tree crowns and vehicles cluster. We will use CanopyRS/SelvaBox, which has 14 rasters tiled into more than 2,400 images with over 83,000 boxes. The setting tests whether the idea is about "memory of where objects are" in general, or only about temporal coherence.

Baselines

The naive reference points bracket the compute range. Cheap everywhere applies bilinear upsampling and the cheap pass to all patches, which sets the accuracy floor at minimum cost. Heavy everywhere refines all 64 patches every frame, which sets the accuracy ceiling at maximum cost. Uniform random $k$ picks $k$ patches at random per frame, which checks whether any selector beats chance at the same budget.

The strongest existing method is DPR's per-frame learned patch selector (Patch-based Selection and Refinement for Early Object Detection). On FBDD, the BDD100K subset of objects smaller than 1.5% of the image, it reports mAP 4.33 against 2.38 for SR3, and a 77.2% reduction in refined data volume. It is the strongest competitor because it targets exactly this regime with a trained selector. For fairness, DPR's selector will be retrained on our split. It will then be run in top-$k$ mode at every budget $k$ we test, as well as at its native threshold, with the same refiner, the same cheap pass, and the same hyperparameter-trial budget as our scheduler.

The frame-level cascade baseline is Adaptive Feeding (Adaptive Feeding: Achieving Fast and Accurate Detections by Adaptively Combining Object Detectors). It routes whole images to a fast or a slow detector and reports a 50% speedup over SSD500 alone with no loss in mAP on PASCAL VOC 2007. We will reimplement it with our cheap pass and our "heavy everywhere" configuration as its two detectors, then tune its cost-sensitive threshold to match our average compute.

Two ablations isolate the key ingredient. EMA + top-$k$ keeps an exponential moving average of refined confidence per patch and deterministically picks the top $k$ by the product of memory and heuristic. This version has the same memory but no stochasticity and no floor. EMA + ε-greedy adds uniform exploration and so gives a floor without ant-style proportional sampling. We also remove the memory entirely by setting $\rho=1$, and we remove the floor by setting $\tau_{\min}=0$.

Beating DPR means a higher tiny-object mAP at matched mean refined-patch fraction, and a lower 95th-percentile time-to-first-detection, across at least three budgets. The comparison uses three seeds, with paired bootstrap confidence intervals over sequences. No reported result was found on Lacuna for any temporal or stateful patch scheduler on these benchmarks, so we will establish the video numbers for all baselines ourselves.

Why not just…?

Why not just keep an exponential moving average of detections and refine the top patches? This is the real threat. Our pheromone update is an exponential moving average, and the ant-style sampling may add nothing beyond an ε-greedy floor. The two ablations above test this directly. If EMA + ε-greedy matches our scheduler on both accuracy and time-to-first-detection, we will report it: the contribution then becomes "bounded temporal memory for patch allocation," and the colony vocabulary is dropped as unnecessary. The ant-specific pieces that could still matter are proportional sampling, which spreads the budget across several plausible patches instead of piling onto one, the exponential deposit rule, and neighbor diffusion. Each will be switched off separately.

Why not just lower DPR's selector threshold so it misses less? A lower threshold buys recall with compute, and it still carries no information from one frame to the next. An object the selector cannot see in the downsampled frame stays invisible whatever the threshold. The matched-budget top-$k$ sweep of DPR's selector traces its full trade-off curve. Our claim fails if that curve dominates ours at every budget.

Why not just route whole frames, as Adaptive Feeding does? Frame-level routing cannot exploit the fact that most of each frame is empty. DPR's 22.8% object-patch figure suggests most refinement spent on a "hard" frame is wasted. Adaptive Feeding also runs its proposal network on every frame before routing. We still include it at matched compute, because on scenes crowded with objects, frame-level routing may win, and the note will report where the crossover lies.

Why not just search a cheaper detector architecture with ants, or with gradient-based search? That is the obvious intersection, and it is poorly placed. EAutoDet: Efficient Architecture Search for Object Detection already searches backbone and feature pyramid jointly in 1.4 GPU-days and reports 40.1 mAP at 120 FPS on COCO. DetOFA: Efficient Training of Once-for-All Networks for Object Detection using Path Filter cuts supernet search cost by 30–63%. Meanwhile, DeepSwarm's own scaling result, 7 to 18 hours for a 0.13% error gain, suggests ant search would struggle to compete. Static search also makes every frame uniformly cheaper, whereas our question is where to spend compute within a frame. The two are complementary, and we will run an EAutoDet-s-style small detector as an alternative cheap pass to confirm that the scheduler's gains persist on top of a searched detector.

Proposed Experiments

The main experiment runs on BDD100K driving sequences, restricted to FBDD-style tiny objects. It assumes enough frame-level annotation density; if that is lacking, we will use the annotated keyframes and propagate labels with a tracker, marking those frames as auxiliary. We sweep the budget $k$ from 4 to 32 of 64 patches. For each scheduler we report tiny-object mAP against the mean refined-patch fraction, measured end-to-end latency on one embedded-class GPU, and the distribution of frames from an object's first annotation to its first correct detection.

The decisive check has two parts. First, the scheduler should beat DPR's selector at matched compute in the 95th-percentile time-to-first-detection. Second, the measured tail of that distribution should sit near the predicted $1/q$ bound, and the no-floor ablation should show a long or unbounded tail. If the first holds but the second does not, the bound is not doing the work we claim.

A second check injects synthetic objects into empty regions of real sequences at known frames. This measures recovery from a guaranteed miss without depending on annotation density. The SelvaBox tiling experiment then tests whether spatial rather than temporal memory transfers.

Finally, we report sensitivity maps over $\rho$, $\alpha/\beta$ and $\tau_{\min}$. If good settings form a narrow ridge, the method inherits the tuning fragility DeepSwarm documented, which would weaken the practical case even if the peak numbers are strong.