When capturing a scene with high contrast—such as a dark room with a bright window—a single camera exposure is rarely enough. A short exposure captures the details outside the window but leaves the room in pitch black; a long exposure reveals the room but turns the window into a featureless white blob. The goal of multi-exposure fusion is to combine these frames into a single High Dynamic Range (HDR) image that contains the best of both worlds.
The challenge is that these source images are fundamentally inconsistent. A pixel that is "correct" in one frame is "broken" (saturated or noisy) in another. Traditional fusion methods often struggle with these contradictions, leading to ghosting artifacts or washed-out details. Modern research has converged on a powerful solution: using spatial and channel-wise attention to act as an intelligent gating mechanism.
Attention as a Spatial Gate
The core intuition behind this research direction is that the network should not treat all input pixels equally. Instead, it should learn to "look" only at the parts of each frame that contain valid information.
In ADNet, Zhen Liu and colleagues demonstrate that spatial attention can adaptively weight regions across multiple exposures. This allows the system to prioritize high-quality data while ignoring saturated pixels in long-exposure frames and noisy pixels in short-exposure frames. As noted in the NTIRE 2021 Challenge summary by Eduardo Pérez-Pellitero, these attention mechanisms function as spatial gates. They selectively filter out unreliable data, ensuring that the final reconstruction is built only from the most high-fidelity information available in the source stack.
Weighting Features Across Channels
While spatial attention decides where to look, channel-wise attention decides what features are most relevant for reconstruction. In complex scenes, simple pixel-level weighting isn't enough to recover fine textures or structural integrity.
Researchers like Vien Gia An and Chul Lee have pioneered "Exposure-Aware Dynamic Weighted Learning." Their approach uses both spatial and channel-wise attention to merge features from different exposure levels. By embedding the relevance of specific features directly into the fusion process, the network can adaptively select information based on its contextual importance. This is particularly effective for sensors with spatially varying exposures, where the model must "hallucinate" missing data by weighting the reliability of neighboring pixels and feature maps.
Beyond HDR: A General Theory of Fusion
This mechanism of "selective integration" extends beyond just brightness levels. The same logic applies whenever we need to merge disparate sources of information:
- Infrared and Visible Light: In Res2NetFuse, Song Xu and Hui Li use weight maps generated from mathematical operations (like the l1-norm) to identify salient features. This ensures that thermal signatures from infrared images and high-resolution textures from visible images are preserved without blurring each other out.
- Multi-View Synthesis: When generating images of humans in new poses, Enver Sangineto and Aliaksandr Siarohin use local attention to prioritize the most informative source features from multiple viewpoints. This allows the system to bypass occlusions—if one camera view is blocked, the attention mechanism simply "gates" that view out and relies on a clearer one.
Why It Works: Intuitive Selection
The shift from static fusion (like averaging) to attention-guided fusion works because it mirrors how a human might manually edit a photo. If you were masking two photos together in Photoshop, you wouldn't set the opacity to 50% everywhere; you would paint a mask to hide the "blown-out" sky in one layer and reveal the "clean" sky in the other.
By using attention, models like those seen in the NTIRE 2022 Challenge learn to perform this masking automatically and at a much more granular level. Whether it's Keuntek Lee using codebooks to hallucinate details or Zhen Liu using deformable convolutions to align frames before fusion, the underlying principle remains the same: identify the most informative regions and suppress the noise.