Overcoming the Planar Assumption in Image Alignment

This direction is merged into another direction: planar scene assumption. Its last version stays readable and citable here.

Version 1 of 134 papers38 conceptsfrozen 2026-10-04

In computer vision, the homography is a fundamental tool—a $3 \times 3$ matrix that maps points from one image plane to another. It is the mathematical engine behind panorama stitching, document scanning, and basic image registration. However, a growing body of research highlights a critical "blind spot" in this model: the assumption that the world is flat.

The Geometry of a Flat World

A homography accurately describes the relationship between two views of a scene under only two specific conditions: either the scene being imaged is a perfectly flat plane (like a wall or a distant floor), or the camera is rotating around its own center without any translation (zero baseline).

When these conditions are met, every pixel in the first image has a predictable, linear destination in the second. As Jonathon Hare notes in Saliency for Image Description and Retrieval, this geometric simplicity allows for efficient re-ranking and matching. But as soon as a camera moves through a three-dimensional environment—a drone flying through a forest or a robot navigating a cluttered room—this simplicity collapses.

The Parallax Problem

The primary reason homographies fail in 3D environments is parallax. When a camera translates (moves linearly), objects at different depths appear to move at different speeds across the sensor. If you look out a car window, the distant mountains barely move while the nearby fence posts zip past.

A single $3 \times 3$ homography matrix cannot account for these varying speeds. As Zhiying Jiang and Risheng Liu explain in their work on Modality-Invariant Representation, this assumption fails in environments with significant depth variations. In these cases, the "planar model" is no longer a simplification—it is a distortion.

Researchers like Hoang Le have shown in Deep Homography Estimation for Dynamic Scenes that even state-of-the-art deep learning models struggle when scenes aren't static or flat, as the parallax effects violate the underlying geometric assumptions of the transformation.

Why We Still Use It (And How We Fix It)

If homographies are so fragile, why do they remain the standard? The answer lies in their efficiency and the difficulty of the alternative: full 3D reconstruction. Estimating a fundamental matrix or a depth map is computationally expensive and often requires more visual information than is available.

To bridge this gap, researchers are developing "3D-aware" alignment strategies:

  1. Hierarchical and Local Refinement: Instead of one global warp, models like LocalTrans by Ruizhi Shao use local attention kernels to handle non-planar distortions piece-by-piece.
  2. Differentiable Optimization: Che-Han Chang introduced CLKN, which embeds classical Lucas-Kanade layers into neural networks to achieve sub-pixel accuracy even when large motions challenge the planar assumption.
  3. Sensor Fusion: In robotics, the "flat world" assumption is particularly dangerous for navigation. Nitin J. Sanket and Cornelia Fermuller demonstrate in EVDodgeNet and PRGFlow that relying solely on homography for ego-motion estimation fails in complex 3D environments like forests. Their solution involves integrating IMU data and event cameras to maintain stability where geometry alone fails.

The Training and Evaluation Bias

A subtle but pervasive issue identified by Alexander Mai in Training Deep Neural Networks to Detect Repeatable 2D Features is that most of our AI models are trained on synthetic 2D homographies. Because these models never "see" 3D depth during training, they fail to generalize to the real world.

Furthermore, our benchmarks might be hiding these flaws. Karel Lenc and Andrea Vedaldi argue in their evaluation of feature detectors that datasets relying on homography for ground truth are restricted to planar scenes. This creates a circular logic where we optimize for a "flat world" because our tests only measure performance on a "flat world," leaving the complexities of 3D occlusions and non-rigid deformations largely unaddressed.

The path forward, as suggested by the recent work of Yingfu Xu in CUAHN-VIO, involves moving toward uncertainty-aware networks that know when the planar assumption is likely to break, allowing the system to fall back on other sensors or more complex geometric models when the world stops being flat.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_overcoming_the_planar_assumption_in_imag_109038b9,
  title = {Overcoming the Planar Assumption in Image Alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/overcoming-the-planar-assumption-in-image-alignment/rdr_109038b9ee2841c9b25228062460b2d6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_overcoming_the_planar_assumption_in_imag_109038b9_v1,
  title = {Overcoming the Planar Assumption in Image Alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/overcoming-the-planar-assumption-in-image-alignment/rdr_109038b9ee2841c9b25228062460b2d6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}