In computer vision tasks like human pose estimation, face alignment, and temporal action detection, we often represent the location of a point (a joint, a landmark, or a timestamp) as a heatmap. Because of memory and computational constraints, these heatmaps are usually much lower in resolution than the original input.
When we try to extract a single coordinate from these discrete grids, we encounter a fundamental "quantization error." If a network predicts a peak at pixel $(10, 10)$, but the true landmark is at $(10.4, 9.7)$, the standard approach of taking the maximum value (the argmax) fails to capture that precision. Researchers have developed a family of analytical techniques to "look between the pixels" and recover sub-pixel accuracy without increasing the resolution of the neural network itself.
The Gaussian Assumption
The core intuition behind sub-pixel recovery is that a predicted heatmap is not just a collection of independent pixels; it is a discrete sampling of a continuous underlying distribution.
In early work on 3D pipe reconstruction, Atsushi Yamashita and Toru Kaneko modeled the cross-sectional intensity of a laser line as a Gaussian distribution. By fitting this model to the captured pixels, they could estimate the center of the laser stripe with sub-pixel accuracy. Similarly, in biological imaging, Fernando de M. Dossin and Alexandre Dufour used Gaussian sub-pixel localization to pinpoint telomere clusters smaller than the diffraction limit of light microscopy.
If we assume the "true" activation in a heatmap follows a 2D Gaussian distribution, we can use the values of the peak pixel and its immediate neighbors to solve for the true maximum mathematically.
Analytical Peak Estimation
Modern deep learning pipelines have refined this into "model-agnostic" post-processing steps. Ying Tai and Xiaoming Liu introduced Fractional Heatmap Regression (FHR) for face alignment. Instead of relying on the discrete grid, they interpret low-resolution heatmaps as continuous surfaces. By applying a closed-form mathematical solution to the local neighborhood of the peak, they derive sub-pixel coordinates that remain stable even in high-resolution video, reducing the "jitter" often seen in frame-by-frame tracking.
The most prominent implementation of this idea is DARK (Distribution-Aware coordinate Representation for Keypoints), developed by Feng Zhang, Xiatian Zhu, and Hanbin Dai.
DARK replaces heuristic "shifting" rules (like moving the peak by 0.25 pixels toward the second-highest neighbor) with a principled approach using Taylor expansion.
How Taylor Expansion Recovers the Peak
If we treat the heatmap as a continuous function $f(x)$, we can approximate its value near the maximum grid point $m$ using a second-order Taylor expansion:
$$f(x) \approx f(m) + f'(m)(x - m) + \frac{1}{2}f''(m)(x - m)^2$$
To find the "true" sub-pixel maximum, we look for the point where the derivative of this approximation is zero. This allows us to calculate a sub-pixel offset based on the gradient and the Hessian matrix (the second-order derivatives) of the log-heatmap at the discrete peak.
Feng Zhang and colleagues demonstrated that this simple mathematical refinement can boost the accuracy of state-of-the-art models like HRNet by significant margins (up to 5.7% AP on the COCO dataset) without requiring any retraining or architectural changes.
Generalizing to Time and Distributions
The problem of quantization isn't limited to 2D images. In Temporal Action Detection (TAD), models process video in discrete "snippets," which often leads to imprecise start and end times for actions. Sauradip Nag and Xiatian Zhu adapted the Taylor expansion method into GAP (Gaussian Approximated Post-processing). By treating the 1D temporal boundary scores as a distribution, GAP recovers precise timestamps that fall between the fixed intervals of the initial sampling grid.
While DARK and GAP focus on decoding coordinates from heatmaps, Haoxuan Qu and Jun Liu proposed rethinking the encoding process. Instead of forcing the model to predict a fixed Gaussian "blob," they frame pose estimation as a distribution matching problem. By using Earth Mover’s Distance (EMD), they allow the model to learn to distribute "mass" across adjacent grid cells in a way that naturally represents sub-pixel coordinates. This approach treats the ground-truth point as a set of "sub-pixel demanders," forcing the model to achieve high precision by matching the continuous distribution rather than just hitting a discrete pixel target.