To capture the full fidelity of the real world, digital photography often relies on combining multiple frames. High Dynamic Range (HDR) imaging merges exposures tailored for deep shadows and bright highlights, while burst photography averages rapid successions of short exposures to eliminate sensor noise.
However, these techniques rely on a fragile assumption: the static scene constraint. Traditional fusion algorithms assume that the camera and every object in the scene remain perfectly still across the entire capture sequence. When this assumption is violated by a moving vehicle, a walking pedestrian, or simple camera shake, direct pixel-wise merging fails. Instead of a clean composite, the resulting image suffers from ghosting artifacts—translucent, blurry, or duplicated shapes where moving subjects appear overlaid across multiple temporal states.
Frame 1 (Short Exp) ---> [Object at Position A] \
Frame 2 (Medium Exp) ---> [Object at Position B] ----> Direct Merge ----> Ghosted Image (Both A & B visible)
Frame 3 (Long Exp) ---> [Object at Position C] /
Resolving this problem—known as deghosting—requires computational photography frameworks to intelligently reconcile spatial misalignments across frames before or during the fusion process.
Why Simple Alignment Fails
A naive solution to ghosting is to apply standard image registration or optical flow to warp all frames onto a single reference frame. In practice, dynamic multi-exposure sequences present unique challenges that break conventional alignment algorithms:
- Luminance Inconsistency: Standard optical flow algorithms rely on brightness constancy—the assumption that a pixel's intensity does not change as it moves. In multi-exposure imaging, this assumption is fundamentally broken. As shown by Jia et al. in Image Registration with Global and Local Luminance Alignment, algorithms must jointly estimate geometric transformations and complex local luminance mappings to prevent severe registration errors.
- Spatially-Varying Motion Blur: Long-exposure frames naturally suffer from motion blur, whereas short-exposure frames remain sharp but noisy. As Portz et al. demonstrated in Optical flow in the presence of spatially-varying motion blur, attempting to map sharp features directly to motion-blurred regions distorts flow fields, leading to structural artifacts and inaccurate warping.
- Information Occlusion and Saturation: In highly dynamic scenes, moving objects create disocclusions (revealing background regions that were hidden in other frames). Furthermore, as noted by Chen et al. in Attention-Guided Progressive Neural Texture Fusion for High Dynamic Range Image Restoration, a region might be perfectly exposed in one frame but completely saturated (clamped to pure white) or under-exposed (lost to pure black) in another. When the underlying pixel-wise correspondence is discontinuous or missing, explicit geometric warping fails.
The Dual Strategy: Alignment and Rejection
To circumvent the limitations of explicit warping, modern deep learning architectures adopt a dual-branch or progressive strategy. Instead of forcing a perfect geometric alignment everywhere, they divide the workload: align what can be structurally reconciled, and reject or suppress features that introduce ghosting.
A prominent mechanism for implicit alignment is the Deformable Convolutional Network (DCN). Unlike standard convolutions that sample from a fixed grid, a deformable convolution learns spatial offsets for its kernel inputs. This allows the network to dynamically warp feature maps in the latent space, correcting for camera jitter and non-rigid object motion without requiring an explicit optical flow field.
However, alignment alone cannot fix regions where an object has entirely moved away. To handle this, networks introduce spatial attention mechanisms. In the single-authored work ADNet: Attention-Guided Deformable Convolutional Network for High Dynamic Range Imaging, Zhen Liu combined deformable convolutions with a spatial attention module. The attention mechanism computes a pixel-wise confidence map, identifying areas where the content of non-reference frames conflicts with the reference frame. By down-weighting these inconsistent or saturated regions, the network effectively "rejects" the ghosting components, relying solely on the reference frame for structure while extracting exposure details from the aligned regions.
Similarly, dual-branch architectures like DRHDR split the workload spatially or resolutionally: a full-resolution branch preserves sharp details in aligned regions, while a low-resolution branch focuses globally on detecting and suppressing ghosting artifacts.
Context-Awareness and Global Search via Transformers
While convolutional networks excel at local alignment, they struggle when handling large motions or severe occlusions. If an object moves a significant distance across the frame between exposures, the local receptive field of a standard CNN kernel cannot locate the corresponding features.
To address this, researchers have turned to Vision Transformers (ViTs) to leverage their global self-attention mechanisms. Transformers treat deghosting as a global context-matching problem: if a patch of texture is obscured or saturated in the reference frame, the model can look across the entire spatial extent of the non-reference frames to find the missing information.
[Reference Frame: Saturated Highlight] <====== Global Attention ======> [Non-Reference Frame: Sharp Detail]
(Reconstructs missing texture)
To optimize this computationally expensive process, Song et al. introduced Selective TransHDR: Transformer-Based Selective HDR Imaging Using Ghost Region Mask. The authors utilized a dedicated ghost region mask to isolate misaligned zones. Efficient CNNs process the static, well-aligned regions of the image, while the Transformer is selectively deployed only within the masked ghost regions to perform a global search for missing details.
By modeling long-range dependencies, context-aware transformers—such as the one proposed by Zhen Liu et al. in Ghost-free High Dynamic Range Imaging with Context-Aware Transformer—can synthesize clean, ghost-free composites even when faced with large spatial offsets and severe exposure discrepancies.
Alternative Paradigms: Kernel Prediction and Multi-Modal Guidance
Beyond the standard pipeline of feature alignment and attention-based fusion, two alternative paradigms have emerged to tackle the root causes of ghosting.
Spatially Varying Kernel Prediction
Instead of explicitly warping pixels or features, some frameworks predict local, linear reconstruction filters. In Burst Denoising With Kernel Prediction Networks, Mildenhall et al. demonstrated that a network can output a unique per-pixel pooling kernel for a burst of images. When applied to the neighboring frames, these predicted kernels perform alignment and averaging simultaneously. Because the network can output zero-weights for inconsistent pixels, it naturally handles complex local motion and avoids explicit registration errors entirely.
Multi-Modal Sensor Fusion
The fundamental cause of ghosting in video and multi-exposure imaging is the temporal gap between sequential captures. To eliminate this gap, recent work explores multi-modal guidance. Han et al. developed a framework detailed in Learning Event Guided High Dynamic Range Video Reconstruction that pairs standard low dynamic range frames with data from event cameras. Event sensors measure continuous changes in intensity at microsecond resolutions. By aligning the frames within a shared latent space guided by this continuous event stream, the network gains precise motion cues, allowing it to reconstruct high-speed HDR video completely free of ghosting and motion artifacts.