Zero Baseline Coaxial Imaging for Multimodal Fusion

This direction is split into other directions: coaxial optical alignment. Its last version stays readable and citable here.

Version 1 of 144 papers51 conceptsfrozen 2026-10-04

In computer vision, combining different types of data—like heat, motion, and color—usually starts with a geometric headache. If you place a thermal camera next to an RGB camera, they see the world from slightly different perspectives. This "parallax" means that a pixel at $(x, y)$ in one image doesn't correspond to $(x, y)$ in the other. To fuse them, software must warp the images, a process that requires knowing the depth of every object in the scene.

This creates a frustrating circular dependency: to align the images, you need depth; but to calculate high-quality depth, you often need the aligned images. Researchers are increasingly breaking this loop by moving the solution from software into hardware, using coaxial alignment to ensure multiple sensors share the exact same optical axis.

The Geometry of Zero Baseline

The traditional approach to multi-sensor systems is the "stereo rig," where sensors are mounted side-by-side. As Chenglong Li and colleagues note in their work on RGB-T tracking, even parallel axes require complex registration. The "baseline"—the physical distance between the lenses—is the enemy of perfect fusion.

Coaxial imaging eliminates this baseline entirely. By placing a beam splitter or a specialized prism behind a primary lens, the incoming light is divided. One portion of the light travels to an RGB sensor, while the other is reflected toward a different modality, such as an infrared or event-based sensor.

As Hanyu Zhou demonstrates in the CoSEC dataset, this "zero-baseline" configuration ensures that every sensor captures the scene from an identical perspective. There is no parallax to correct, no warping artifacts, and no computational overhead for registration. The fusion becomes "pixel-perfect" by design.

Bridging Disparate Modalities

This hardware-level alignment is particularly transformative when the sensors "see" the world in fundamentally different ways.

  • Event Cameras and RGB: Event cameras capture microsecond-level changes in brightness but lack color and texture. By using a beam splitter to align an event camera with a standard RGB sensor, Kaizhen Sun and Peiqi Duan have created benchmarks like EvTTC and EventAid. These systems allow researchers to use the high temporal resolution of events to "de-blur" RGB frames or handle extreme lighting, knowing that the event and the pixel are perfectly co-registered.
  • Infrared and Visible Light: Nigel Morris and Shai Avidan used a gold dichroic beam-splitter to study the statistics of infrared images, finding that while textures differ, the underlying structural "DNA" is shared. More recently, Jinnyeong Kim and Seung-Hwan Baek developed a prism-based system that achieves pixel-aligned RGB-NIR imaging, significantly improving robot vision in low-light "edge cases."
  • Active Sensing (Lidar and Lasers): The concept extends beyond passive cameras. Mark Sheinin and his team at CMU developed a system that projects thermal patterns using a laser aligned coaxially with a thermal camera. Because the projector and camera share an axis, every laser ray maps directly to a camera pixel, simplifying 3D reconstruction on textureless surfaces.

Precision in High-Stakes Environments

The most compelling argument for coaxial hardware is found in fields where registration errors aren't just a nuisance, but a danger.

In medical imaging, Nassir Navab and his team used a double-mirror system to align an optical camera with an X-ray source. This allows the optical feed to act as a radiation-free proxy for the X-ray path, letting surgeons position equipment without unnecessary radiation exposure. Similarly, Peter A. von Niederhäusern demonstrated that aligning optical sensors with gamma ray collimators allows for the direct projection of radiation data onto a visible image, helping surgeons locate lymph nodes with "augmented reality" precision.

Why It Works: The Death of the Correspondence Problem

The intuition behind this research direction is the elimination of the correspondence problem. In a standard multi-sensor setup, the system must constantly ask: "Which point in Image A corresponds to this point in Image B?" This question is computationally expensive and prone to failure when objects are close to the camera or when textures are sparse.

By using hardware like beam splitters (Runzhao Yang) or shared-path periscopes (Zhiqian Su), researchers ensure the answer is always "the same pixel index." This allows neural networks to focus on the content of the fusion—such as identifying a tumor or a pedestrian in the dark—rather than the geometry of the alignment. As the field moves toward more complex multi-modal models, the "zero-baseline" approach provides the cleanest possible data foundation.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_zero_baseline_coaxial_imaging_for_multim_ef8364f8,
  title = {Zero Baseline Coaxial Imaging for Multimodal Fusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-baseline-coaxial-imaging-for-multimodal-fusion/rdr_ef8364f80885446da1a1549da366aa21},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_zero_baseline_coaxial_imaging_for_multim_ef8364f8_v1,
  title = {Zero Baseline Coaxial Imaging for Multimodal Fusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-baseline-coaxial-imaging-for-multimodal-fusion/rdr_ef8364f80885446da1a1549da366aa21/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}