Flow matching trains a neural network to predict a velocity field, meaning a direction in which to move each noisy image so that, integrated over time, noise turns into data. The same model also defines an exact likelihood, but reading it out requires the divergence of the velocity field at every step. The divergence is the sum of how much each output coordinate changes when its own input coordinate is nudged. For a $32 \times 32 \times 3$ image this is the trace of a $3072 \times 3072$ Jacobian. Computing it exactly takes thousands of backward passes, so practice falls back on random-probe (Hutchinson) estimates, which are unbiased but noisy. The noise enters both likelihood evaluation and newer training objectives that match divergences directly. The difficulty comes from the network's receptive field, the set of input pixels that can influence a given output. Modern velocity networks have global receptive fields, so every output depends on every input, the Jacobian is dense, and nothing structural makes its diagonal easy to extract.
We propose to split the velocity network into two parts, so that global receptive field and exact divergence are no longer in conflict. The first part is a global context network that is provably blind to the pixel it describes: its output at pixel $i$ depends on every pixel except $i$. The second is a tiny per-pixel head that combines pixel $i$'s own channels with that context. Because the context carries no dependence on $x_i$, the diagonal of the full Jacobian comes only from the head. The exact divergence of the whole image then costs $C$ cheap Jacobian-vector products through the head, with $C = 3$ for RGB, regardless of resolution. This transfers the "hollow Jacobian" idea from particle systems to image grids. It also removes the locality restriction that limited the particle version, by routing the blind context along rows and columns across the full image.
Core Claim
Write the velocity at pixel $i$ and time $t$ as
$$v_t(x)_i = \tau_\theta\big(x_i,\ c_\phi(x_{-i}, t)_i,\ t\big),$$
where $x_i \in \mathbb{R}^C$ is the pixel's channel vector, $x_{-i}$ is every other pixel, $c_\phi$ is a global context network constrained so that its output at $i$ never depends on $x_i$, and $\tau_\theta$ is a small per-pixel network. The divergence is then exact and local:
$$\mathrm{div}\, v_t(x) = \sum_{i} \mathrm{Tr}\!\left(\frac{\partial \tau_\theta}{\partial x_i}\right)\Big|_{c \text{ held fixed}}.$$
The context is computed once. A handful of forward-mode passes through $\tau_\theta$ then give all $C \times C$ diagonal blocks at once.
The parameterization loses no expressive power in principle, because a rich enough $c_\phi(x_{-i})$ can encode all of $x_{-i}$, and $\tau_\theta$ then sees the whole image. What remains open is whether practical blind-context architectures can match unconstrained U-Nets on sample quality. The main experiment tests exactly this.
The claim is threefold. First, this construction gives exact per-sample image likelihoods from a flow matching model at a cost of $C$ passes rather than $D$. Second, it makes divergence-matching training exact rather than stochastic. Third, both hold without giving up the global receptive field that the local-receptive-field literature shows matters.
Where This Sits
The directly motivating work is HollowFlow (also here; HollowFlow: Efficient Sample Likelihood Evaluation using Hollow Message Passing). It builds Boltzmann-generator velocity fields whose Jacobian has zero diagonal blocks, via non-backtracking message passing on particle graphs. The trace then costs $d$ backward passes instead of $n \times d$, with a reported 100× speedup on LJ55. HollowFlow supplies the core trick. It is also limited by its reliance on $k$-nearest-neighbor graphs, which the file flags as restricting long-range interactions. We change the domain from particles to pixels, where the per-site state is the channel vector. We also replace local neighbor graphs with blind routing that covers the whole image.
The second motivating paper is Improving Flow Matching by Aligning Flow Divergence. It shows that conditional flow matching leaves a "divergence gap" between learned and true probability paths, and closes it with a divergence-matching loss estimated by Hutchinson probes. That loss improves CIFAR-10 NLL from 2.99 to 2.85 and FID from 6.35 to 5.62. Our construction makes that loss's divergence term exact. This removes a source of gradient noise the paper had to live with.
Several papers constrain the design by showing what fails. Exact-likelihood flows with global context exist in discrete form. Generative Flows with Invertible Attentions (AttnFlow) puts attention inside checkerboard coupling layers and reaches FID 33.6 on CIFAR-10. Invertible Attention places Lipschitz-bounded attention in residual flows and reaches 3.39 bits/dim, versus 3.45 for Invertible Residual Networks. Both show that global context helps flows. Both also sit far behind flow matching on sample quality. Invertible Attention in addition inherits truncated power-series log-determinants and quadratic memory. The Convolution Exponential gets its log-determinant from the trace of a local kernel, but only because its support is local; it expands the receptive field by about 12 pixels per layer. Together these say that cheap Jacobian quantities have so far been bought with locality or with discrete-flow restrictions. Our proposal gets cheapness from a blind-spot constraint instead.
The routing method comes from the receptive-field literature. CCNet shows that two passes of row-plus-column attention reach every pixel at $\mathcal{O}(N\sqrt{N})$ cost. Multi-Dimensional Recurrent Neural Networks show that directional sweeps across a 2D grid give each site context from whole quadrants. MintNet shows how masked convolutions enforce strict directional dependence. We combine these to build context that is global yet provably excludes one pixel. Neither CCNet nor the multi-dimensional RNNs has any notion of this exclusion, because they were not built for density estimation.
Research Program
Blind context network. It has four branches.
The "up" branch alternates two kinds of layer. The first is full within-row mixing, such as row attention or a wide row convolution. The second is column mixing that is causal in the row index. A final one-row shift ensures row $a$ sees only rows strictly above it. The "down" branch mirrors this.
The "left" branch runs on pixels of row $a$ only. Its inputs are each pixel's own values concatenated with that pixel's up and down contexts. It aggregates strictly causally in the column index. The "right" branch mirrors it.
Because the up and down contexts at $(a, b')$ never touch row $a$, and the left branch at $(a,b)$ sees only columns $b' < b$, the union covers every pixel except $(a,b)$. The per-layer cost matches criss-cross attention. The blindness property is structural. We will also verify it numerically by checking that $\partial c_i / \partial x_i = 0$ to machine precision.
Per-pixel head. It is a small MLP on $[x_i, c_i, t]$. We will also test a variant whose context is blind to a $3 \times 3$ neighborhood and whose head is a single $3 \times 3$ convolution followed by pointwise layers. In that variant the exact diagonal is recovered by structured probing: pixels at least three apart share a probe, so $9C = 27$ passes suffice. This measures how much a slightly wider local path buys.
Training. We use standard conditional flow matching. We then add the divergence-matching loss of the divergence-alignment paper, with the divergence computed exactly by our method instead of by Hutchinson probes.
Baselines
The naive reference point is an unconstrained flow matching U-Net whose likelihood is computed by brute-force exact trace: $D$ backward passes per function evaluation on a subset of test images. This gives ground-truth likelihoods and the cost we aim to avoid.
The strongest existing method is Flow and Divergence Matching from Improving Flow Matching by Aligning Flow Divergence. It reports 2.85 NLL and FID 5.62 on CIFAR-10, against 2.99 NLL and FID 6.35 for plain flow matching. We will reproduce both in our codebase with the same backbone width, training steps, optimizer, and ODE solver, and use them as the comparison for our model. We match parameter count and training FLOPs, counting the extra cost of our context branches. We give each method the same learning-rate and schedule sweep.
Beating the strongest baseline means one of two outcomes. The first is equal or better NLL with FID within about 0.3 of the reproduced FDM number, at a fraction of FDM's likelihood-evaluation cost. The second is lower NLL than FDM because exact divergence training outperforms stochastic divergence training.
We also compare against discrete exact-likelihood flows with global context, using reported numbers: AttnFlow (FID 33.6), and Invertible Attention (3.39 bits/dim) with its i-ResNet baseline (3.45).
Two ablations isolate the key ingredients. The first removes global reach by replacing the four-branch context with a blind context of small fixed radius. If the local-receptive-field literature is right, FID should degrade clearly. The second removes blindness by using the same architecture without masks and Hutchinson divergence. This measures what the structural constraint costs in sample quality.
Why not just…?
Why not just use Hutchinson with more probes? Hutchinson is unbiased, so for a dataset-averaged bits/dim a single probe per image already works. This is a real threat to any claim about aggregate NLL, and we will not make one on that basis. The case for exactness rests on per-sample quantities and on training. Estimator variance falls only as one over the number of probes, and its size scales with the off-diagonal mass of a dense Jacobian. We will plot per-image likelihood error against probe count and wall-clock, using the brute-force trace as reference, and ask how many probes match our $C$-pass cost. For training, the matched comparison is FDM with Hutchinson versus FDM with exact divergence. If the two give the same NLL and FID, the training benefit is not supported, and the contribution narrows to cheap exact evaluation.
Why not just use a local velocity network, whose banded Jacobian already allows exact trace by structured probing? That works, but the probe count grows with the square of the receptive-field width, and receptive fields in useful U-Nets span the whole image. Keeping the network local enough for cheap probing reintroduces the long-range failures documented by AttnFlow and CCNet. The small-radius ablation tests this directly. The $3 \times 3$ head variant is the compromise: local where the diagonal lives, global everywhere else.
Why not just use a discrete flow with invertible attention, which already has exact likelihood? The file's numbers answer this. AttnFlow reaches FID 33.6 on CIFAR-10, and Invertible Attention's residual flows rely on truncated, stochastic log-determinants and quadratic memory. Flow matching models sit near FID 6. The question is whether exact likelihood can be had at flow matching quality, which these designs do not deliver.
Proposed Experiments
The first check is exactness. On MNIST and on $16 \times 16$ CIFAR crops, we compare our divergence to the brute-force $D$-pass trace at many random states and times. We expect agreement to floating-point tolerance, and we will confirm the zero self-dependence of the context numerically.
The second, decisive for the whole proposal, is sample quality on CIFAR-10 under matched compute. We report FID and bits/dim for plain flow matching, reproduced FDM, the blind-spot model, and the two ablations. If the blind-spot constraint costs more than about one FID point relative to the unconstrained U-Net, we treat the parameterization as practically restrictive despite its theoretical universality. In that case the $3 \times 3$ variant becomes the proposal.
The third tests exact versus stochastic divergence training. We train FDM with exact divergence from our architecture and with Hutchinson divergence on the same architecture with masks removed. We compare NLL, FID, and the variance of the loss gradient.
The fourth measures cost. We compare likelihood-evaluation wall-clock and per-image likelihood error against Hutchinson at matched budgets, and we test scaling with resolution, where our cost should stay flat in $C$.
Finally, as a stress test of larger dimension, we extend blindness to space-time and apply the model to KTH video prediction. The divergence-alignment paper reports FVD 155.5 for FDM versus 180 for flow matching on this task. We check whether exact divergence holds that quality when $D$ is far larger. No reported per-sample likelihood results were found on Lacuna for any flow matching image model, so the per-sample comparisons will rely entirely on our brute-force references.
References
- HollowFlow: Efficient Sample Likelihood Evaluation using Hollow Message Passing
- Improving Flow Matching by Aligning Flow Divergence
- Generative Flows with Invertible Attentions
- Invertible Attention
- Invertible Residual Networks
- The Convolution Exponential and Generalized Sylvester Flows
- CCNet: Criss-Cross Attention for Semantic Segmentation
- Multi-dimensional Recurrent Neural Networks
- MintNet: Building Invertible Neural Networks with Masked Convolutions