Standard deep learning architectures are built on rigid, discrete grids. Convolutional kernels look at fixed pixel neighborhoods, pooling layers downsample by integer factors, and transformers attend to distinct token indices. However, many real-world problems require neural networks to operate on continuous spatial coordinates—whether they are learning to crop an image, warp a feature map, deform a convolutional kernel, or place physical sensors in a room.
The fundamental challenge is that discrete lookups are non-differentiable. If a network predicts a bounding box coordinate or a sensor location as a floating-point number, rounding it to the nearest integer grid cell creates a step function. The derivative of a step function is zero everywhere and undefined at the transitions, completely breaking backpropagation.
Differentiable spatial sampling solves this problem. By replacing discrete lookups with continuous interpolation kernels, it transforms spatial coordinates into learnable parameters that can be optimized end-to-end using standard gradient descent.
The Core Mechanism: Why Interpolation Enables Gradients
To understand why differentiable sampling works, consider a one-dimensional feature map $F$ and a continuous coordinate $x$ where we want to sample a value. If we simply round $x$ to the nearest integer index, the sampled value does not change when $x$ varies slightly, yielding a gradient of zero.
Instead, we can find the two closest integer coordinates, $x_0 = \lfloor x \rfloor$ and $x_1 = x_0 + 1$, and compute a weighted average based on the fractional distance $\alpha = x - x_0$:
$$F(x) = (1 - \alpha)F[x_0] + \alpha F[x_1]$$
Because the weight $\alpha$ is a continuous function of $x$, we can now compute the partial derivative of the sampled value with respect to the coordinate $x$:
$$\frac{\partial F(x)}{\partial x} = F[x_1] - F[x_0]$$
This simple difference is the magic key. It provides a directional signal to the network: if the downstream loss function wants the sampled value $F(x)$ to increase, and $F[x_1]$ is larger than $F[x_0]$, the gradient will push $x$ toward $x_1$. If $F[x_1]$ is smaller, it pushes $x$ toward $x_0$.
Discrete Lookup (Rounding):
F(x) ───┐
└─────── Gradient is ZERO everywhere
Differentiable Sampling (Linear Interpolation):
F(x) ───╱─── Gradient is the SLOPE (F[x_1] - F[x_0])
In two dimensions, this concept scales to bilinear interpolation, where a continuous coordinate $(x, y)$ samples from its four nearest neighbors. As formalized by Max Jaderberg et al. in their seminal paper on Spatial Transformer Networks, this mechanism allows a network to learn explicit geometric transformations—such as scaling, rotation, or affine warps—without any explicit coordinate supervision.
Deforming Kernels and Attention Windows
Once spatial coordinates become differentiable, we are no longer restricted to uniform grid operations. We can allow the network to dynamically reshape its own receptive fields based on the input data.
Learnable Convolutions
Standard convolutions apply weights to a rigid rectangular grid. In contrast, Active Convolution: Learning the Shape of Convolution for Image Classification by Yunho Jeon et al. treats the spatial offsets of the kernel weights as continuous parameters. By using bilinear interpolation to sample from these fractional offsets, the kernel can custom-tailor its shape during training to better capture non-rigid structures.
Similarly, Ismail Khalfaoui Hassani et al. introduced Dilated convolution with learnable spacings, which allows a small number of weights to move freely across a massive receptive field, achieving the representational capacity of large kernels with a fraction of the parameters.
However, standard bilinear interpolation can sometimes introduce biased gradient fields that hinder convergence. To address this, Yu Liu et al. proposed Differentiable Kernel Evolution, introducing a "gradient agreement" interpolation function that ensures partial derivatives point directly toward reference points, stabilizing the optimization of kernel shapes.
Adaptive Sequence Modeling
This continuous relaxation extends naturally to 1D domains like text and time series. In Transformable Convolutional Neural Network for Text Classification, the authors utilize 1D linear interpolation to allow convolutional filters to span non-consecutive words, effectively capturing long-range linguistic dependencies like "not...good".
For transformers, rigid attention patterns can be computationally wasteful. The authors of Adaptive Attention for Sparse-based Long-sequence Transformer use linear interpolation to sample information from continuous token positions, allowing the model to dynamically optimize where its attention points land. This concept is mirrored in time-series forecasting by DeformableTST: Transformer for Time Series Forecasting without Over-reliance on Patching, where the authors use linear interpolation to let the network discover and track critical historical time points without relying on rigid, predefined window patches.
End-to-End Localization and View Synthesis
By placing a differentiable sampling layer between a coordinate predictor and a target generation task, downstream losses can directly guide spatial localization.
┌──────────────────┐ Coordinates ┌────────────────────────┐
│ Bounding Box / │────────────────────>│ Differentiable │
│ Offset Predictor │ │ Spatial Sampling Layer │
└──────────────────┘ └────────────────────────┘
▲ │
│ ▼
│ Gradients Flow Back ┌────────────────────────┐
└────────────────────────────────────│ Downstream Task / Loss │
└────────────────────────┘
Dense Captioning and Cropping
In DenseCap: Fully Convolutional Localization Networks for Dense Captioning, Justin Johnson et al. implemented a differentiable cropping layer using bilinear interpolation. This architectural choice allows language generation gradients to flow all the way back to the region proposal mechanism, meaning the model learns to adjust its bounding box coordinates based on the semantic richness of the text description it is generating.
This capability is also leveraged for unsupervised or weakly supervised tasks. For instance, the authors of Repurposing existing deep networks for caption and aesthetic-guided image cropping optimize crop parameters directly against pre-trained aesthetic and captioning networks, using the gradient flow to find the most visually pleasing bounding box without explicit bounding box training data.
View Synthesis via Appearance Flow
Instead of forcing a network to generate pixels from scratch—which often results in blurry artifacts—Tinghui Zhou et al. demonstrated in View Synthesis by Appearance Flow that a network can instead predict a coordinate offset field. By using a differentiable sampling layer to move pixels from the source image to the predicted coordinates, the model retains sharp, realistic textures while learning complex 3D perspective shifts entirely through backpropagation.
Relaxing Discrete Optimization and Physical Constraints
The utility of differentiable spatial sampling extends far beyond standard computer vision; it serves as a powerful mathematical proxy for solving discrete placement and physical optimization problems.
Hardware and Sensor Placement
In chip design, placing components on a 2D grid to minimize routing congestion is traditionally a non-differentiable combinatorial problem. The authors of RoutePlacer: An End-to-End Routability-Aware Placer with Graph Neural Network bypass this constraint by using a soft assignment function to distribute a component's influence across neighboring grid bins. This transforms the discrete mapping into a continuous, differentiable proxy that can be optimized via standard backpropagation.
A similar strategy applies to physics-informed neural networks. In PhySense: Sensor Placement Optimization for Accurate Physics Sensing, the authors treat physical sensor coordinates as continuous, learnable parameters within a differentiable reconstruction pipeline. The gradients of the reconstruction loss directly guide the sensors to move toward locations that yield the maximum information gain.
Continuous Functional Fields
Differentiable sampling also allows neural networks to interface directly with continuous physics simulations and partial differential equations (PDEs). As demonstrated by Anadi Chaman in FunkNN: Neural Interpolation for Functional Generation, embedding differentiable interpolation kernels within a spatial transformer module allows a network to compute exact analytical gradients and Laplacians at any arbitrary point in a generated field. This eliminates the need for less accurate finite-difference approximations, enabling the direct solution of inverse problems and PDEs within a deep learning framework.
By transforming spatial selection from a hard, discrete choice into a smooth, continuous landscape, differentiable spatial sampling provides the mathematical bridge necessary for neural networks to reason about, deform, and optimize the physical and geometric world.
Key Foundational Work
View Synthesis by Appearance Flow (2016) — contributed 1 concept elements to this direction.
Spatial Transformer Networks (2015) — contributed 1 concept elements to this direction.
DenseCap: Fully Convolutional Localization Networks for Dense Captioning (2016) — contributed 1 concept elements to this direction.