Constraint Aggregation: Fixing the Aperture Problem in Global Motion Attention

When you watch a moving edge through a small window, you can only tell how fast it moves perpendicular to itself. Sliding along the edge changes nothing in the image. This is the aperture problem. Modern optical flow networks, which predict a motion vector for every pixel between two video frames, face it constantly. Their early features see only small neighborhoods, so along edges and in weakly textured regions they measure one component of motion, not two. The accepted remedy is to let each pixel borrow motion from far away. Global Motion Aggregation (GMA) does this with self-attention over the whole image. Each pixel takes a weighted average of motion features from pixels that look similar to it. That averaging is exactly the operation that biological and classical vision tell us gives the wrong answer for aperture-limited signals. Averaging the perpendicular motions of two differently oriented edges on the same rigid object gives a velocity that is too slow, and for asymmetric configurations it points in the wrong direction. The correct combination is the intersection of constraints: each edge says "the true velocity lies on this line," and the answer is where the lines meet.

We propose to change what global attention aggregates in a flow network. Instead of mixing motion vectors or opaque motion features, each pixel contributes a small uncertainty-shaped constraint. This is a 2×2 matrix saying which directions of motion it has measured and how reliably, plus the matching measurement. Attention sums these constraints and solves a 2×2 linear system per pixel. Edges then contribute only along their normal. Corners contribute in both directions. Distant evidence combines by geometry rather than by blending. The module drops into a recurrent all-pairs architecture of the RAFT family at negligible cost. We will test whether this structural change, rather than more receptive field, is what closes the remaining gap on aperture-dominated regions.

Core Claim

Let pixel $j$ carry a positive semidefinite information matrix $A_j \in \mathbb{R}^{2\times 2}$ and an information vector $b_j = A_j u_j$, where $u_j$ is its local flow estimate. For a pure edge with unit normal $n_j$, $A_j \approx \sigma_j n_j n_j^\top$ is rank one, with confidence $\sigma_j$. For a corner it is full rank. Given attention weights $w_{ij}$ from appearance similarity, as in GMA, we estimate the aggregated flow at pixel $i$ as

$$\hat v_i = \Big(\sum_j w_{ij} A_j + \lambda I\Big)^{-1} \sum_j w_{ij}\, b_j ,$$

where $\lambda$ is a small damping constant and $I$ is the identity. This is the least-squares intersection of the attended constraints, the same fusion used in classical normal-flow estimation such as Best Linear Unbiased Estimation for 2D and 3D Flow with Event-based Cameras. There, though, the support set is fixed by hand rather than chosen by learned attention. The claim is that GMA-style vector aggregation cannot represent this operation with scalar weights. Our matrix-valued aggregation can. The gain should therefore concentrate on pixels whose local structure is rank deficient, and it should show up as a measurable drop in error there even when overall receptive field is held constant.

How This Sits Relative to Prior Work

Several papers motivate the proposal. Towards Understanding Adversarial Robustness of Optical Flow Networks identifies the aperture problem combined with small encoder receptive fields as the root of patch vulnerability. It fixes this by deepening the encoder of FlowNetC, which reduces attacked error from 52.66 to 9.60 EPE. That establishes receptive field as a causal factor, but it only enlarges the field and never changes how evidence is combined. How Do Neural Networks Estimate Optical Flow? finds that the FlowNetS decoder "fills in" ambiguous surfaces from confident edges and corners. That suggests networks do discover constraint propagation implicitly, but it is an analysis only. Modeling Human Visual Motion Processing with Trainable Motion Energy Sensing and a Self-attention Network shows that self-attention over V1-like motion energy reproduces barber-pole illusions and human-perceived Sintel flow (partial correlation 0.57 vs. 0.20 for RAFT). It targets perceptual alignment, not geometric accuracy, and its attention still pools vectors.

Two papers constrain the design. Learning Optical Flow with Kernel Patch Attention (KPA) argues that global attention imports noisy, irrelevant context, and it reaches strong numbers with local-only attention. We take this seriously. Our reading is that global context is noisy partly because it is blended as vectors. A distant pixel whose constraint is orthogonal to the query's ambiguity adds information. One that is parallel adds nothing and, in matrix form, cannot bias the unmeasured direction. High-Resolution Optical Flow from 1D Attention and Correlation shows that factorizing the search into axis-aligned 1D passes costs accuracy (Sintel Clean 1.98 vs. RAFT's 1.43). For that reason we keep the attention support two-dimensional in the main study and treat criss-cross factorization (CCNet) only as an efficiency variant.

The method borrows from two sources. Good Edgels to Track uses the image structure tensor to measure local directional degeneracy. That tensor is a 2×2 matrix of summed gradient outer products, which separates edges (one strong direction) from corners (two). Our $A_j$ is its learned analogue, but we replace the epipolar assumption, which fails for independently moving objects, with attention-selected support. RAFT (RAFT) supplies the backbone and its correlation lookup.

Research Program

We will build two versions of the constraint module on top of the GMA codebase, replacing only the aggregation step. In the analytic version, $A_j$ comes from fitting a quadratic to the RAFT correlation lookup window around the current flow estimate at each recurrent iteration. The negated Hessian of the matching score, projected to be positive semidefinite, becomes $A_j$, and the Newton step defines $u_j$. This is the Lucas–Kanade structure tensor computed on learned features. In the learned version, a small head predicts a Cholesky factor of $A_j$ from correlation and context features, trained end to end with the usual flow loss. Both versions output $\hat v_i$ together with the eigenvalues of the aggregated matrix, a per-pixel certainty. These are concatenated into the recurrent update, replacing GMA's aggregated motion features. The update block, loss, and training schedule are unchanged.

We will also build a diagnostic dataset in which the correct answer requires intersection of constraints. It will contain moving plaids with asymmetric components, where vector averaging and intersection of constraints predict different directions. It will also contain rigidly translating and rotating outlined polygons on textureless backgrounds, and barber-pole apertures. Rendering these at FlyingThings resolution with ground-truth flow lets us compute error separately on edge pixels, corner pixels, and interior pixels. We will then label Sintel and KITTI pixels by the rank of their image structure tensor (edge, corner, flat) and report error per class. Occlusion masks will be reported alongside, because GMA's gains are concentrated there.

Baselines

The naive reference is RAFT itself, trained on the standard FlyingChairs → FlyingThings3D → Sintel/KITTI (with HD1K) schedule. The file quotes it at Sintel Clean 1.43 EPE (reported in the Flow1D paper) and at 5.10% F1-all on KITTI-15 (reported in KPA). It shows what a shallow encoder plus recurrent local refinement achieves with no global aggregation.

The strongest competitor on this specific question is GMA (Learning to Estimate Hidden Motions with Global Motion Aggregation). It is the method our module modifies. It reports a 13.6% (Final) and 13.7% (Clean) AEPE improvement over RAFT; no reported result found on Lacuna for GMA's absolute Sintel or KITTI numbers. We will retrain it from the released configuration under our exact schedule, iteration count, and seeds. Beating it means lower EPE on rank-deficient (edge and flat) Sintel pixels with at least parity overall. We need three seeds, and the edge-pixel gap must exceed seed variance.

The strongest published numbers in the file are KPA's: 1.35 Clean and 2.36 Final EPE on Sintel, and 4.60% F1-all on KITTI-15. We will add KPA's local module to both GMA and our model to test complementarity. We claim a win over KPA only if our module, alone or combined with it, improves on 4.60% F1-all under the same training.

Two ablations isolate the key ingredient. The first replaces $A_j$ with a scalar confidence times the identity. This is confidence-weighted vector averaging, with the same parameters and compute. The second freezes $A_j = I$, which recovers plain attention-weighted averaging of flow estimates. If the scalar ablation matches the full model, the matrix structure is not doing the work, and the claim fails.

Why not just…?

Why not just deepen or widen the encoder before the correlation layer? The adversarial-robustness study shows that a larger pre-correlation receptive field helps a great deal on FlowNetC. It is a real threat, because RAFT's encoder is shallow. A larger field makes each pixel's feature more distinctive, but it still produces a single match per pixel. Along a long straight edge, a larger window sees more of the same edge and stays rank deficient. We will test this directly by training a RAFT+GMA variant with a deeper encoder of matched parameter count. On the outlined-polygon diagnostic, where edges are longer than any plausible receptive field, we predict it fails where constraint aggregation succeeds. If it does not fail there, we will report that receptive field suffices.

Why not just use local patch attention like KPA? Local smoothing does propagate motion from a corner along its attached edge, and KPA's numbers are excellent. When the nearest corner lies outside the window, as on large textureless objects, no local evidence exists. GMA's occlusion gains show that this case matters on Sintel. Our per-structure-class breakdown on Sintel, and the polygon diagnostic with object sizes swept beyond KPA's patch size, will show where local attention stops working.

Why not just add a scalar confidence to GMA's attention? This is the simplest competitor and the one most likely to capture much of any gain. A scalar can down-weight edges but cannot say which direction an edge constrains. For a query on a vertical edge, it cannot prefer a horizontal edge's evidence over another vertical edge's. The asymmetric-plaid diagnostic has a closed-form wrong answer for any scalar-weighted average, so the comparison is decisive rather than a matter of tuning. That is why this is our central ablation.

Why not rely on the recurrent update to learn intersection of constraints implicitly? It might. The FlowNetS analysis suggests networks learn filling-in. If the recurrent unit of plain GMA already solves the plaid diagnostic after training, our explicit solve adds nothing, and the experiment will show it.

Decisive Checks

The first check is the diagnostic set. Plain GMA and the scalar ablation should show direction errors on asymmetric plaids matching the vector-average prediction. The constraint model should track the intersection-of-constraints prediction. The second check is Sintel Clean and Final and KITTI-15 under identical training, with error split by structure-tensor class and occlusion. The predicted signature is gains concentrated on edge and flat pixels, not uniformly spread. The third check is the human-perceived Sintel subset used by the motion-energy paper. That subset probes whether geometric constraint fusion moves perception-alignment correlation above RAFT's 0.20, a secondary but informative test. The fourth check reruns the patch-attack protocol of the adversarial study on RAFT+GMA and our model. A patch produces confident but false constraints. Matrix fusion should limit their influence to the directions they actually constrain, so error outside the patch should grow less than under GMA.

If the gains appear only in the analytic version, the Hessian of the correlation surface is the useful signal. If they appear only in the learned version, the network needs freedom to shape uncertainty. Either result tells us how deep flow networks should represent aperture ambiguity. A natural follow-on applies the same module to event-camera normal flow, whose inputs are rank-one constraints by construction (Learning Normal Flow Directly From Event Neighborhoods).