Where Do Tiny Objects Die in Manifold-Prior Restoration?

A popular way to restore a degraded image, for example to undo 4× downsampling, is to learn a generator whose outputs trace out a low-dimensional surface of plausible images, then search that surface for the image that best explains the measurement. Injective normalizing flows suit this job. They are invertible networks that map a small latent vector one-to-one into the large image space, so they give an exact projection onto the learned surface and an exact likelihood over it. Such models are usually trained in two stages: first fit the surface by reconstruction, then fit a probability density on it. Nobody has checked what this recipe does to tiny structures, such as 8–16 pixel aerial targets, crop pests, and small defects. These are exactly the details downsampling destroys and a restoration method would most need to bring back.

We propose to use the one thing injective, two-stage models uniquely provide, a clean split between "not representable" and "representable but improbable", to find out where tiny objects are lost. We then fix the stage responsible. Our hypothesis is that the loss happens almost entirely in the first stage. The reconstruction loss weights errors by pixel area, and a single global latent code cannot spend capacity on small objects that can appear anywhere. So no density model or restoration algorithm downstream can recover them. The fix we test is a scale-split injective generator. A global branch produces the coarse image. A shared, position-independent local branch produces fine detail patch by patch. Each branch has its own reconstruction objective in the first stage.

Core Claim

Trumpets: Injective Flows for Inference and Inverse Problems already combines all three ingredients at the obvious level. It is an injective flow, trained with a reconstruction ("MSE") phase and then a likelihood phase, and used as a prior for 4× super-resolution on CelebA. There it reaches 20.07 dB SNR against Deep Image Prior's 14.12 dB. It also reports that its samples are smoother than GAN outputs, and it treats resolution loss only as an external linear operator applied to faces, which contain no tiny objects. Our proposal therefore does not claim "injective flows for super-resolution". It makes two narrower claims.

The first is diagnostic. For a ground-truth high-resolution image $x$, the exact projection $\Pi(x)$ onto the learned surface asks a question with no measurement noise involved: can the model represent this image at all? If a tiny object in $x$ is absent from $\Pi(x)$, no prior-based restoration using this model can produce it. We will call that a manifold failure. If the object survives projection but disappears in the restored estimate, the cause lies in the density prior or in genuine ambiguity of the measurement. We will call that a density failure. Two-stage injective models make this split exact, because the projection is a closed-form or cheap least-squares inverse (Universal Joint Approximation of Manifolds and Densities by Simple Injective Flows gives one for injective ReLU layers). Bijective flows, VAEs and GANs cannot offer this split. We predict that for objects below roughly 16×16 pixels, manifold failures dominate.

The second claim is constructive. If manifold failure dominates, then dividing latent capacity across spatial scales, with a local chart shared across positions, should recover tiny objects better than a single global chart with the same total latent dimension. It should also beat simply reweighting the reconstruction loss.

The Scale-Split Injective Generator

We write the image through an orthogonal Haar wavelet transform. This splits it into a low-pass band (a coarse, downsampled copy of the image) and detail bands (fine edges and texture). The generator is built from two parts. A global injective flow $g$, of Trumpets type, maps a global latent $z_0 \in \mathbb{R}^{d_0}$ to the low-pass band. A local injective flow $h$, with weights shared across all patches, maps each patch latent $z_p \in \mathbb{R}^{k}$ to the detail coefficients of patch $p$, conditioned on the local low-pass context:

$$x = W^{-1}\Big[\, g(z_0),\ \big\{ h(z_p;\ c_p(g(z_0))) \big\}_{p=1}^{P} \Big],$$

Here $W^{-1}$ is the inverse wavelet transform, $P$ is the number of patches, and $c_p$ extracts the coarse context around patch $p$. The dependence runs one way, from coarse to fine, and each branch is injective in its own latent. The whole map is therefore injective, and projection reduces to a coarse inverse followed by patch-wise inverses. The total latent dimension is $d = d_0 + Pk$. We only need densities in latent space, as in the two-step scheme of Diagnosing and Fixing Manifold Overfitting in Deep Generative Models. That paper proves reconstruction-then-latent-density recovers the data distribution. So we do not need the ambient volume factor.

Training has two stages. In the first stage, $g$ is fit by mean squared error on the low-pass band. $h$ is fit by mean squared error on detail patches, with patches sampled uniformly over the image rather than weighted by area. This keeps a 1%-of-pixels object from being drowned out; the file notes pests average about 1.58% of box area in LLPD-26. In the second stage, a bijective flow fits the density of $z_0$, and a second flow, conditioned on context, fits the density of the patch latents.

Research Program

We first build the diagnostic and run it on a plain Trumpets model trained on aerial crops. The procedure is to downsample, then project and restore with the method of Trumpets: Injective Flows for Inference and Inverse Problems, and then count, per ground-truth tiny object, whether it is detected in the original image, in the projection $\Pi(x)$, and in the restored estimate. A fixed detector trained on native-resolution data makes these calls. This yields a manifold-versus-density attribution curve as a function of object size.

Second, we train the scale-split generator and its controls at matched total latent dimension and matched compute, and repeat the attribution. Third, we address the objection from Conformal Embedding Flows: Tractable Density Estimation on Learned Manifolds. That paper found sequential training far worse than joint training on CelebA (FID 171 vs 118), because a frozen manifold cannot adapt to the density model. We run a joint-trained variant of each generator. If our reading is right, joint training will improve global sample quality but will not rescue tiny-object survival under projection, because the joint objective is still dominated by area. That result would reconcile the two-stage camp (Diagnosing and Enhancing VAE Models, Flows for simultaneous manifold learning and density estimation) with the conformal-flow result. Those works measured global quality, and the tiny-object failure is a separate axis.

Feasibility is supported by Conditional Injective Flows for Bayesian Imaging, which trains injective flows on 256×256 images on one GPU in under a day. That is the crop size we use.

Baselines

The naive reference point runs the fixed detector directly on the 4×-downsampled image after bicubic upsampling, with no learned prior. Every restoration method must beat it on tiny-object average precision to justify itself.

The strongest existing method on the restoration side is Trumpets with its iFlow-L restoration algorithm (Trumpets: Injective Flows for Inference and Inverse Problems). Its reported number is 20.07 dB SNR for 4× super-resolution on CelebA 64×64. We first reproduce this number on CelebA to validate our implementation. We then train it on the same aerial crops, with the same total latent dimension $d$, GPU-hour budget and hyperparameter search budget as our model. Beating it means higher tiny-object survival under projection and higher tiny-object average precision after restoration, without losing more than a fraction of a dB of whole-image SNR.

On the detection side, the strongest reference is the flow-based tiny-object detector of Noise-Robust Tiny Object Localization with Flows. It reports 24.4% AP on AI-TOD with DINO at native resolution. This is an upper reference for what a detector achieves with full-resolution input, not a competitor on low-resolution input. We report how much of the gap between the naive reference and this ceiling each restoration method closes. We also train a detector directly on low-resolution images with learned importance pooling from LIP: Local Importance-Based Pooling, which raised COCO small-object AP from 21.9% to 24.0%. No reported result was found on Lacuna for LIP on 4×-downsampled AI-TOD, so we run it ourselves with the same training budget.

Two ablations isolate our key ingredient. The first is a single global injective flow with the same total latent dimension $d$, trained with the same patch-uniform, band-separated reconstruction loss. This separates architecture from loss. The second is our scale-split model with the local branch's weights not shared across positions. This tests whether a chart that works the same at every position is what matters.

Why not just…?

Why not just increase the latent dimension of a single Trumpets model? Tiny objects might vanish simply because $d$ is too small. Every comparison is therefore made at matched $d$, and we also sweep $d$ for the global-only model. If the global model's tiny-object survival rises to ours at comparable $d$, the scale split is unnecessary and we will say so. We expect it will not. A global chart must spend separate coordinates on the same object at every possible location, while the shared local chart spends $k$ dimensions per patch once.

Why not just reweight the first-stage loss toward detail bands or object boxes? This is the most serious threat. It directly attacks the area-weighting we blame. The global-only ablation with our exact loss tests it. If that ablation matches the scale-split model, the finding becomes "the manifold stage fails because of how the loss is allocated, not because of architecture". That is still a clean, publishable diagnosis, and we would report it as the main result rather than argue around it.

Why not just train jointly, as the conformal-flow results suggest? Joint training beat sequential training on CelebA FID in Conformal Embedding Flows: Tractable Density Estimation on Learned Manifolds. Faces, however, have no tiny objects, and FID measures global realism. Our joint-trained variants measure the effect on survival under projection directly. If joint training fixes tiny objects, the two-stage design is the wrong lever.

Why not skip generative restoration and fix the detector? Low-resolution detector training with learned pooling is included as a baseline. It may well win on average precision alone. It cannot say whether an object is absent from the image or merely unrecoverable, and it gives no projection distance or likelihood per object. If the detector wins by a wide margin, the case for manifold priors in this regime weakens, and we will report that.

Proposed Experiments

The main benchmark is AI-TOD, whose average object size is 12.8 pixels. We use 256×256 crops downsampled 4× to 64×64 with additive noise. LLPD-26 serves as a second domain with sub-2% objects. CelebA 4× super-resolution is used only for replication.

Metrics are computed per object-size bin (<8, 8–16, 16–32 pixels). They are tiny-object survival rate under exact projection, tiny-object average precision on restored images from the fixed detector, SNR inside object boxes and over the whole image, and the manifold-versus-density attribution fractions.

The decisive check is the projection test on ground-truth high-resolution images. If the global Trumpets model already preserves most sub-16-pixel objects under projection, our central hypothesis is false. In that case density failure dominates, and the work redirects to fixing the second stage. If the global model erases them, the loss-reweighted global model still erases many at matched $d$, and the scale-split model preserves them, the architectural claim stands.

A second check fixes the restoration algorithm's prior weight at zero, which leaves only data fit on the manifold. It confirms that object loss seen after projection is not an artifact of the prior. A third check verifies that the improvement in tiny-object average precision does not come from hallucination: we measure false positives on crops containing no objects.