Hybrid Sensor Fusion for High Spatiotemporal Resolution

This direction is merged into another direction: spatiotemporal resolution trade-off (temporal-spatial resolution trade-off). Its last version stays readable and citable here.

Version 1 of 118 papers18 conceptsfrozen 2026-10-04

Digital imaging sensors face a fundamental hardware bottleneck: you can have high spatial resolution or high temporal resolution, but rarely both. Capturing a high-resolution image requires a long exposure time to gather enough light for a clean signal, which leads to motion blur if anything moves. Conversely, capturing at high frame rates requires short exposures and fast readouts, which forces sensors to reduce pixel counts to manage bandwidth and noise.

To break this trade-off, researchers have developed hybrid imaging systems that fuse data from multiple sensors. By combining a "slow" high-resolution sensor with a "fast" low-resolution or event-based sensor, these systems reconstruct video that is both sharp in detail and smooth in motion.

The Best of Both Worlds

The core intuition behind hybrid sensing is that spatial and temporal information are often redundant. A scene's high-frequency textures (the "what") usually stay consistent, while its motion (the "where" and "when") can be tracked with much less spatial detail.

In A hybrid camera for motion deblurring and depth map super-resolution, Feng Li 0005, Jingyi Yu, and Jinxiang Chai demonstrated this by pairing two low-resolution high-speed (LRHS) cameras with one high-resolution low-speed (HRLS) camera. The LRHS cameras track the motion trajectories, which are then used to "undo" the motion blur in the HRLS image.

This concept was later refined into deep learning frameworks like AWnet. Developed by Ming Cheng, Zhan Ma, and M. Salman Asif, AWnet synthesizes high-resolution, high-frame-rate video by fusing a sharp, slow-rate stream with a blurry, high-rate stream. The network learns to extract the structural details from the sharp frames and propagate them along the motion paths captured by the high-speed frames.

Computational Shuttering and Flexible Voxels

Instead of using two physical cameras, some researchers modify how a single sensor collects light over time. This is known as computational photography.

In Flexible Voxels for Motion-Aware Videography, Mohit Gupta and Amit K. Agrawal introduced a per-pixel shuttering scheme. By varying the exposure timing across the sensor, they allow the user to trade off spatial and temporal resolution after the video is captured. This allows a single capture to produce high-resolution images of static backgrounds and blur-free footage of fast-moving objects simultaneously.

A more theoretical approach is the Coded Aperture Keyed Exposure (CAKE) paradigm. Zachary T. Harmany, Rebecca Willett, and Roummel F. Marcia proved in Spatio-temporal Compressed Sensing with Coded Apertures and Keyed Exposures that combining spatial light modulation with rapid temporal shuttering satisfies the Restricted Isometry Property. This provides a mathematical guarantee that high-fidelity video can be reconstructed from a very small number of compressed measurements.

Event-Enhanced Recovery

The most recent shift in this field involves event cameras—bio-inspired sensors that only report per-pixel changes in brightness rather than full frames. Event cameras have microsecond temporal resolution and massive dynamic range but lack traditional spatial texture.

Bishan Wang and Lei Yu developed eSL-Net, an Event Enhanced High-Quality Image Recovery framework. By "unfolding" a sparse learning algorithm into a neural network, eSL-Net uses the high-speed "events" to guide the deblurring and super-resolution of standard frames.

Taking this further, HR-INR by Yunfan LU and Bin Yang uses Implicit Neural Representations (INR) to achieve continuous space-time super-resolution. Because the event stream is essentially continuous in time, the INR can query the video at any arbitrary timestamp or resolution, effectively capturing non-linear motions that traditional frame-based interpolation would miss.

Applications Beyond Consumer Video

This hybrid approach isn't just for better slow-motion video; it enables scientific measurements that were previously impossible.

By treating spatial and temporal resolution as a joint optimization problem rather than a hardware limit, hybrid sensor fusion allows us to see the world with a level of detail and speed that no single sensor could provide.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hybrid_sensor_fusion_for_high_spatiotemp_02605e75,
  title = {Hybrid Sensor Fusion for High Spatiotemporal Resolution},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-sensor-fusion-for-high-spatiotemporal-resolution/rdr_02605e75c19644449368bf98013a3a9c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_hybrid_sensor_fusion_for_high_spatiotemp_02605e75_v1,
  title = {Hybrid Sensor Fusion for High Spatiotemporal Resolution},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-sensor-fusion-for-high-spatiotemporal-resolution/rdr_02605e75c19644449368bf98013a3a9c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}