Motion Artifacts in Sequential Multi-Frame Imaging

22 concepts18 papers1 proposal

Many advanced 3D sensing and computational photography techniques rely on a fundamental trade-off: to extract high-fidelity information about a scene, they capture a sequence of images rather than just one. Whether it is varying the phase of a light signal, the pattern of a projector, or the focus of a lens, these methods multiplex data across time.

However, this temporal multiplexing rests on a fragile assumption: the scene must remain perfectly static during the acquisition cycle. When objects move, this assumption collapses, leading to temporal multiplexing artifacts—geometric distortions, ghosting, and depth inaccuracies that occur because the sub-frames no longer align to the same physical points in space.

The Static Scene Assumption

In ideal conditions, a sensor captures multiple "looks" at a scene to solve for a specific property. For example, indirect Time-of-Flight (iToF) sensors typically require four sequential phase measurements to calculate a single depth frame. As Meng et al. et al. explain, this multi-step process creates a temporal gap. If an object moves during this interval, the phase calculations—which rely on comparing these four measurements—become mathematically corrupted.

This problem is not unique to ToF. Achar et al. et al. note that light transport analysis, which separates direct and global illumination, requires multiple patterns that necessitate a static scene. Similarly, Qian et al. et al. show that even simple flash/no-flash photography fails when motion occurs between captures, as the subtraction used to isolate the flash signal produces "ghosts" where the subject moved.

Mechanisms of Distortion

When the static assumption is violated, the resulting artifacts manifest in several distinct ways:

  • Ghosting and Blurring: In continuous-wave ToF, rapid motion results in "ghosting" because the sensor integrates light from different spatial positions into the same phase calculation. Attal et al. et al. and Okunev et al. et al. demonstrate that this is particularly severe when motion occurs along the camera's optical axis, leading to significant depth errors.
  • Spatial Misalignment: In structured light systems, where patterns are projected onto a scene, Je et al. et al. observe that high-speed motion causes temporal inconsistency between frames, leading to reconstruction failures. Saragadam et al. et al. highlight that even alternating between patterned and ambient light can create discrepancies that ruin the depth map.
  • Jagged Edges: For multi-frequency ToF, which uses different modulation frequencies to resolve depth ambiguity, Wang et al. et al. identify that the temporal delay between frequency captures causes jagged edges and "flying pixels" at the boundaries of moving objects.

Moving Toward Snapshot Solutions

To overcome these artifacts, researchers generally follow two paths: compressing the sequence into a single "snapshot" or explicitly modeling the motion during reconstruction.

Snapshot Acquisition The most direct way to eliminate temporal artifacts is to avoid sequential capture entirely. Friday et al. et al. proposed "Snapshot Lidar," which uses a rolling shutter and holographic mathematical tricks to capture depth in a single frame, effectively boosting temporal resolution by 4x. In the domain of passive sensing, Luo and Guo developed a "Focal Split" camera that mimics a jumping spider's eye to capture differential defocus in one snapshot, removing the motion vulnerability inherent in traditional focus-sweeping methods like those described by Nagahara et al. et al.

Motion-Aware Reconstruction If multiple frames must be used, the system must account for the scene's dynamics. Wang et al. et al. utilize large-displacement optical flow to align phases before fusion. More recently, Meng et al. et al. showed that synthesizing missing measurement phases for every timestamp allows for high-frame-rate depth imaging that is inherently resistant to ghosting. Taking this further, Okunev et al. et al. introduced F-TöRF, which uses a 4D volumetric radiance field and a differentiable simulator to "undo" the motion corruption, allowing for high-fidelity reconstruction even when objects move faster than the camera's raw frame rate.

By shifting from the assumption of a static world to one that explicitly accounts for time and motion, these methods allow high-precision computational imaging to function in the messy, dynamic environments of the real world.

Key Foundational Work

High-Contrast Color-Stripe Pattern for Rapid Structured-Light Range Imaging (2004) — contributed 1 concept elements to this direction.
Capturing multiple illumination conditions using time and color multiplexing (2009) — contributed 1 concept elements to this direction.
Two-Wavelength Single-Shot Interferometry (2009) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 18 papers
  2. Version 1 · 2026-10-04 · 17 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_temporal_multiplexing_artifacts_a536012c,
  title = {temporal multiplexing artifacts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-multiplexing-artifacts/rdr_a536012c23cd469191a759f4b97a6321},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_temporal_multiplexing_artifacts_a536012c_v2,
  title = {temporal multiplexing artifacts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-multiplexing-artifacts/rdr_a536012c23cd469191a759f4b97a6321/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake