One-Step Sampled Sub-Pixel Keypoints for Low-Resolution Inputs

Most keypoint and tiny-object localizers work in two stages. A coarse head picks a grid cell on a downsampled feature map. A small regression head then adds a fractional offset that moves the point off the grid. This design is fast, but the offset head outputs a single vector. When the input is low resolution (a distant person, a 12-pixel vehicle in an aerial image, a blurred patch), the true position given the features is often ambiguous. It may be spread over several plausible locations, and those locations are correlated across the keypoints of one object. A head trained with an L1 or L2 loss responds by predicting the average of the plausible answers. That average can fall between two good candidates, can form an anatomically impossible pose, and carries no signal about how much to trust it. Benchmarks that score with hard thresholds penalize exactly this failure: a pose metric that counts a keypoint as correct only within a small radius, matching accuracy within 1 pixel, or box precision on tiny targets.

We propose replacing the single-vector offset head with a small generative head. It draws samples of the joint sub-pixel offsets of all keypoints of an instance in a single network evaluation, using the average-velocity flow formulation of Mean Flows for One-step Generative Modeling. Many samples run in parallel as one batch. We pick a mode of the samples as the prediction and use their spread as a calibrated error estimate. The claim is narrow and testable. As input resolution drops, a mode-seeking one-step sampler should beat mean regression on thresholded localization metrics. It should match a many-step flow head at roughly the cost of one regression pass. And its advantage should come from modeling a joint, multimodal distribution, which simpler per-keypoint density heads cannot do.

Core Claim

The research file frames sub-pixel offset regression, resolution degradation, and iterative sampling latency as separate problems. We think they meet at one mechanism. Downsampling turns the offset target from a near-deterministic quantity into a broad, structured conditional distribution. Generative models capture such distributions but are slow because they integrate over many steps. Average-velocity flows remove most of that cost.

The proposal therefore treats resolution degradation as the controlled variable that creates the problem. Sub-pixel offset regression is the task. Iterative latency is the constraint that rules out off-the-shelf diffusion heads.

Concretely, let $c$ be the backbone features pooled at the coarse cells chosen for an instance with $K$ keypoints. Let $z \in \mathbb{R}^{2K}$ be the joint offsets in units of the stride $s$, so that each keypoint lands at $(u + \Delta x)\,s,\ (v + \Delta y)\,s$ for its coarse cell $(u, v)$. We allow $|\Delta| $ to span several cells rather than one, so the head is not confined to a fixed window. We interpolate $z_t = (1-t)\,z + t\,\epsilon$ with Gaussian noise $\epsilon$ and train a network $u_\theta(z_t, r, t \mid c)$ to predict the average velocity over the interval $[r, t]$. Following MeanFlow, the regression target is

$$u_{\text{tgt}} = v - (t - r)\left(v \cdot \partial_z u_\theta + \partial_t u_\theta\right), \qquad v = \epsilon - z,$$

where the bracketed total derivative is computed with one Jacobian-vector product and the target is held fixed during backpropagation. At inference a sample is produced by a single evaluation,

$$\hat z = \epsilon - u_\theta(\epsilon, 0, 1 \mid c),$$

and $N$ such samples are drawn as one batch. The prediction is the highest-density sample under a small kernel density estimate over the $N$ samples. Uncertainty is the spread of those samples.

The head sees only a pooled feature vector and a $2K$-dimensional state. The roughly 16% JVP training overhead reported by MeanFlow therefore applies only to a tiny network, and the backbone cost is unchanged.

Relation to Prior Work

The single-vector offset design this proposal replaces comes from three groups of work. Towards Accurate Multi-person Pose Estimation in the Wild combines heatmaps with short-range offset votes. Stacked Multi-Target Network for Robust Facial Landmark Localisation and Exploiting Offset-guided Network for Pose Estimation and Tracking add residual offsets to heatmap peaks. Pixel-In-Pixel Net pairs grid classification with local offsets on low-resolution maps and runs at hundreds of frames per second. All of them regress one offset per keypoint, which is the conditional mean under an L2 loss. We keep their coarse-cell structure and low-resolution efficiency and change only what the offset head outputs.

Learning to Make Keypoints Sub-pixel Accurate refines detector keypoints with a soft-argmax over a $5\times 5$ or $11\times 11$ patch. Its soft-argmax is a mean over the patch, and it fails when the coarse error leaves the window. Our head is not window-bounded and selects modes instead of averaging.

Two papers motivate the claim that low resolution makes the target distribution non-trivial. Noise-Robust Tiny Object Localization with Flows (TOLF) shows that localization errors on tiny objects are skewed and multimodal. It uses a RealNVP flow only to reweight training gradients, while still predicting a point estimate. We sample from the predicted distribution instead. Detecting tiny objects in aerial images: A normalized Wasserstein distance and a new benchmark shows that a 3-pixel shift drops a tiny box's IoU from 0.54 to 0.14. That is exactly the thresholded regime in which mean-versus-mode matters.

Three papers constrain the design. DiffusionDet shows that a generative localizer run in one step reaches 30 FPS but gives up the accuracy of multi-step sampling. A Gift from the Integration of Discriminative and Diffusion-based Generative Learning recovers fine boundaries with diffusion but names its iterative sampling cost as the main limitation. Together these rule out naive step-dropping. Fast and Accurate Blind Flexible Docking shows that six rounds of deterministic coordinate updates can beat diffusion at 208× lower latency. That result is the most serious argument that sampling is unnecessary, and we test it directly.

MeanFlow supplies the method. It was evaluated only on image and point-cloud synthesis, with 3D MeanFlow covering point clouds. To our knowledge from the file, it has not been used as a conditional localization head.

Research Program

The first stage is a controlled study of the phenomenon itself. We take COCO person keypoints and render each crop at a sweep of effective resolutions. We feed the result to a fixed ResNet-50 backbone with output stride 16 or 32. We first train a well-tuned single-vector offset head. We then measure how often its errors are better explained as "between modes" than as "near one wrong mode," using a many-step flow-matching head as an oracle sampler. If the conditional distributions are nearly unimodal at every resolution, the proposal's premise fails, and we stop and report that.

The second stage builds the one-step joint head and its training recipe. This includes the $(r, t)$ sampling distribution, the size of the offset range in cells, and the number of samples $N$. It also includes mode selection by kernel density versus mean-shift.

The third stage transfers the head to tiny-object localization on AI-TOD. There the "instance" is a center plus a width and height, and the head runs on top of a standard detector.

The fourth stage is a smaller check on keypoint matching on HPatches. There the head replaces the patch soft-argmax as a refiner for SuperPoint keypoints.

Baselines

The naive reference is argmax over a stride-$s$ heatmap with no offset. It shows the size of the quantization ceiling at each resolution.

The direct competitor is a single-vector offset head in the style of Pixel-In-Pixel Net. It uses the same backbone, the same coarse classifier, the same extended offset range, and an L1 loss. Beating it means higher keypoint AP at the lowest two resolutions of the sweep, with no loss at full resolution.

A deterministic iterative refiner tests whether iteration alone explains any gain. It follows the FABFlex pattern: the same head architecture is applied for six rounds, each round predicting a residual update.

Two density baselines test whether any distributional head is enough. The first is a mixture-density head with up to eight Gaussian components over the joint $2K$ offsets. The second is a conditional RealNVP over the same joint vector, modeled on the flow TOLF uses. Both get the same mode-selection rule.

A many-step flow-matching head with the same network serves as the accuracy upper bound and the latency reference. Beating the iterative baselines means matching the many-step head within noise at one evaluation per sample.

Two ablations isolate the key ingredients. The first keeps the MeanFlow head but reports the mean of its samples. This separates mode-seeking from architecture. The second trains independent per-keypoint heads. This separates joint modeling from per-keypoint multimodality.

On COCO keypoints with synthetic downsampling, no reported result was found on Lacuna, so all numbers will be our own reproductions. On AI-TOD, the strongest reported comparison is TOLF on DINO at 24.4% AP (Noise-Robust Tiny Object Localization with Flows). On AI-TOD-v2, DetectoRS with NWD-RKA reports 24.7 AP (Detecting tiny objects in aerial images). On HPatches, the sub-pixel refiner raises 1-pixel matching accuracy from 27% to 33% at about 7 ms per 2048 matches (Learning to Make Keypoints Sub-pixel Accurate).

Every method shares the backbone, training schedule, and data augmentation, and each gets an equal hyperparameter search budget. We will rerun TOLF and the patch refiner in our own pipeline rather than rely only on their published numbers. Latency is measured end-to-end on the same GPU, with $N$ samples counted as one batched call.

Why not just…?

Why not just keep features at high resolution, for example with an HRNet-style network or deconvolution layers? That attacks the cause rather than the symptom. FCPose and Pixel-In-Pixel Net both report that dropping heavy upsampling saves substantial memory and latency without losing precision. More importantly, our degradation sweep lowers the resolution of the input image, and no amount of feature upsampling restores pixels that were never captured. We will still include a high-resolution-feature variant at the full-resolution end of the sweep to quantify the trade-off.

Why not just run a many-step flow or diffusion head, or DiffusionDet with one step? DiffusionDet reports a clear accuracy drop at one step. MeanFlow reports that naive one-step consistency methods lag badly on ImageNet: a 1-step FID of 34.24 for iCT-XL and 10.60 for Shortcut-XL, versus 3.43 for MeanFlow. The many-step head is in our baseline set, so the experiment directly shows whether one-step sampling gives up accuracy.

Why not just predict a Gaussian mixture or a small conditional flow over the offsets? This is the most serious threat. For a single 2D offset we expect a mixture head to work about as well, which is why the proposal models the joint $2K$-dimensional offsets of an instance. In that space, multimodality is structured, for example a whole limb flipped or a whole pose shifted, and fixed-component mixtures are known to struggle. If the joint mixture or RealNVP head ties the MeanFlow head, we will report that the gain comes from mode-seeking over a joint density, not from average-velocity transport. That would still be a useful finding, and the per-keypoint ablation shows which side of this line the result falls on.

Why not just refine deterministically over a few rounds, as FABFlex does? Deterministic refinement is cheap. However, each round still regresses toward a conditional mean, so it should converge to the same between-mode answer when the evidence is ambiguous. The six-round baseline tests this. If it closes the gap at low resolution, the premise that ambiguity rather than under-fitting causes the errors is wrong.

Proposed Experiments and Decisive Checks

The central check is the resolution sweep on COCO keypoints. The prediction is that the AP gap between the MeanFlow head (with mode selection) and the single-vector head is near zero at full resolution and grows as resolution drops. It should also exceed the gap achieved by the mean-of-samples ablation and by the six-round deterministic refiner.

A second check is calibration. Across instances, the spread of the samples should rank-correlate with actual keypoint error. We will compare its ability to flag high-error predictions, measured as area under the precision-recall curve, against the mixture head's variance.

For tiny objects, we train on the original AI-TOD labels and evaluate on the corrected AI-TOD-v2 annotations, for images present in both releases. This turns the dataset's known label jitter into a direct test of whether sampling plus mode selection is more robust than TOLF's gradient down-weighting.

On HPatches, the head should match or exceed 33% accuracy at 1 pixel. It should also recover keypoints whose coarse error exceeds the $11\times 11$ window, which we will count separately.

Latency is reported in every setting. The proposal fails on its own terms if one batched call of $N$ samples costs more than the six-round refiner, or if the many-step head beats it by more than run-to-run noise.