Flow-Free Dual-Camera Demoiréing with Epipolar Gated Cross-Attention

When you photograph a screen with a phone, the display's grid of tiny red, green and blue emitters interferes with the camera sensor's own pixel grid. The result is moiré: rainbow-colored wavy bands that look like part of the image and are hard to remove afterwards. A clean way around this is optical rather than computational. A second, deliberately out-of-focus camera blurs the scene before the light reaches its sensor, so the interfering pattern never forms. That second view holds the true colors and coarse shapes but no fine detail, while the sharp view holds the detail and the moiré. Current dual-camera systems merge the two views by first warping the blurry view onto the sharp one with an optical-flow network, which estimates per-pixel motion, and then stacking the two images as input to a convolutional network. Screens are full of repeated text and grid content, and that is exactly where flow estimation breaks.

We propose replacing the warp-then-stack pipeline with cross-attention, a learned lookup in which each location in the sharp view searches the blurry view for matching content and borrows its colors. The search is restricted to the horizontal line where a rectified side-by-side camera pair must find the match, and a learned gate decides, location by location, how much to trust what it borrowed. The central argument is about repetition. Flow has to commit to a single match. Attention can spread its weight over several equally good matches, and on genuinely repeated content (identical glyphs, identical grid cells) that average is still the correct color. Ambiguity that is fatal for flow is therefore harmless for attention. Where the blurry view is truly wrong, for example at occlusions near depth edges or where both views happen to be in focus, the gate falls back to the sharp view alone.

Core Claim

We claim that on focused–defocused dual-camera demoiréing, a network that aligns the two views implicitly will match or beat the flow-based pipeline when parallax is small, and will degrade much more slowly than it as parallax and content repetition increase. It will also run without a flow network. The alignment comes from cross-attention whose queries come from the sharp view and whose keys and values come from the defocused view, restricted to a one-dimensional band along the line where stereo matches must lie, with a learned blend between borrowed and original features. Both restrictions matter, and we test each one. The band brings attention cost down from quadratic in image area to linear in image width times band height, which is what makes full-HD inputs feasible. The gate keeps the defocused stream from overwriting regions where it carries wrong information.

Positioning Against Prior Work

Two papers establish that the defocused view is a usable clean reference. Self-Adaptively Learning to Demoiré from Focused and Defocused Image Pairs (FDNet) optimizes a restoration network and a blur-kernel network for each image at test time, so that re-blurring the output matches the defocused capture. It reaches 31.77 dB PSNR on SynScreenMoire, but it takes about 30 seconds per image and assumes one blur kernel for the whole image. Video Demoireing using Focused-Defocused Dual-Camera System trains a feed-forward pipeline that warps the defocused frame with FlowFormer, concatenates it with the focused frame inside a multi-scale U-Net, and smooths the result with a joint bilateral filter. It reports 39.23 dB on DualSyntheticVideo against 33.88 dB for DMCNN. These papers motivate our proposal and supply our data. We keep their sensing setup and replace only the alignment-and-fusion step. Neither paper learns which defocused pixel to read from, and neither represents how much to trust it.

Three papers supply the mechanism. Object Tracking by Jointly Exploiting Frame and Event Domain fuses two physically different sensors with bidirectional cross-attention and no explicit warping, and finds plain concatenation insufficient. We borrow the asymmetric query/key structure but make it one-directional, because only the sharp view needs to retrieve information. An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention shows that ungated cross-attention lets one stream wash out the other, and adds a learned gate $g$ that blends attended context with the original features. We reuse that gate. In our setting, though, it has a physical reading: it should close at occlusions and in regions where both views are equally sharp, and we test that prediction directly. DemMamba shows that alignment-free modeling beat an alignment-based method on raw video demoiréing, 30.069 dB against roughly 28.7 dB, while running faster. That is evidence that hard warping is not required. Its setting, however, is consecutive frames from a single camera, not stereo parallax between two lenses.

Two papers constrain the design. The NTIRE 2020 challenge report documents that multi-frame entries scored below single-frame entries when alignment was imperfect. A second input can hurt, which is why the gate and a focused-only fallback are part of the design. Spatial-frequency methods such as FHDe2Net and Spatial-Frequency Attention for Image Denoising process an image's spatial and frequency representations in parallel branches and merge them with convolutions. They show that dense full-image attention is too expensive at full HD. We use one-dimensional band attention instead of large windows for that reason, and we do not pursue spatial-to-frequency cross-attention within a single image, for the reasons given in the "Why not just…?" section.

Research Program

The model has a shared light encoder for both views and a U-Net decoder of the same size as the dual-camera baseline's. At one-quarter and one-eighth resolution we insert the fusion block. Let $q_{x}$ be the query from the focused feature at pixel $x$. Let $k_{x+d}$ and $v_{x+d}$ be the key and value from the defocused feature at horizontal offset $d \in [-D, D]$ within a band of $\pm r$ rows, which absorbs small rectification error. Let $f_x$ be the original focused feature. The fused feature is

$$\hat f_x = g_x \sum_{d} \mathrm{softmax}_d\!\left(\frac{q_x^\top k_{x+d}}{\sqrt{c}}\right) v_{x+d} + (1-g_x) f_x ,$$

where $c$ is the feature dimension and $g_x\in[0,1]$ is predicted from the concatenation of $f_x$ and the attended context. Matching on moiré-contaminated features is a real risk. To reduce it, we compute queries and keys from features of both views after a fixed low-pass filter. The defocused view is already low-pass, so this compares like with like. Values carry the full defocused features. Moiré is aliased low-frequency energy and survives the filter, so we do not expect this to remove contamination. It is a design choice to be tested against unfiltered matching.

Ground truth for parallax-controlled experiments comes from re-rendering. DualSyntheticVideo derives from Sintel, which provides depth, so we can synthesize defocused views at a range of camera baselines (and therefore disparities) and blur radii. The same rendering yields occlusion masks. We also tag regions with high self-similarity, measured by autocorrelation peaks, as "repetitive." We train on DualSynthetic (5,700 pairs) together with these re-rendered variants, using the dual-camera paper's losses: L1, perceptual, FFT high-frequency and adversarial. Matching the losses keeps the comparison about fusion rather than training objectives.

Baselines

The naive reference points come first. One is the unprocessed focused frame. The other is a focused-only network: the same U-Net with the fusion block removed, trained on the same data. This second reference measures what the defocused camera contributes at all. It is the bar that any dual-view result, including the published one, must clear by a clear margin, or the hardware is not worth having.

The strongest existing method is the FlowFormer + concatenation U-Net + joint bilateral filter pipeline of Video Demoireing using Focused-Defocused Dual-Camera System, which reports 39.23 dB PSNR on DualSyntheticVideo. It is the strongest competitor because it is the only trained method in the file that uses the same two-camera input. We will first reproduce its reported number on the original split. We then retrain it on our re-rendered disparity sweep with the same data, losses, optimizer budget and a matched decoder parameter count, using pretrained FlowFormer as in the original. Beating it means two things: matching it within about 0.2 dB at the original disparity, and a significantly smaller PSNR drop at the largest disparities and in repetitive and occluded regions, at lower runtime. We also report FDNet at its published 31.77 dB on SynScreenMoire, run it on the same test pairs with its default iteration count, and record time per image.

Three ablations isolate the key ingredients. The first is concatenation with no alignment, in a U-Net whose receptive field covers the maximum disparity; it tests whether attention is needed at all. The second is the full model with the gate removed ($g\equiv 1$); it tests the gating claim. The third replaces band attention with square-window two-dimensional attention at matched compute; it tests the epipolar restriction. Two secondary variants address specific design choices: matching on unfiltered features, and replacing attention with deformable-convolution alignment.

Why not just…?

Why not just keep flow warping and concatenation, since it already reaches 39.23 dB? This is the main threat. If the disparity in the published setup is small, flow may already be accurate and attention will add nothing beyond speed. The disparity sweep settles the question. If flow + concatenation stays within noise of our model across all baselines and in repetitive regions, the accuracy claim fails, and only the latency argument remains, which we would report as such. Our prediction rests on the repetition argument: on periodic content, flow must pick one of several identical matches, and errors appear as misplaced color.

Why not just concatenate the two views and let a deep CNN with a large receptive field absorb the parallax? For tracking, Object Tracking by Jointly Exploiting Frame and Event Domain found concatenation clearly worse than cross-attention. The NTIRE burst results show unweighted fusion of misaligned inputs can fall below single-frame restoration. Neither result is a demoiréing stereo result, so the unaligned-concatenation ablation is the direct test. We expect it to be competitive at near-zero disparity and to fall behind as disparity grows.

Why not just use deformable convolutions, which learn alignment offsets without a flow network? Deformable convolutions sample a few learned offsets per location. That is the same commit-to-one-match behavior as flow, with fewer samples. We include the variant at matched compute. If it equals band attention on repetitive regions, our repetition argument is wrong, and we will say so.

Why not just skip the second camera and use single-image spatial-frequency attention? That approach avoids hardware. However, the file reports large domain gaps for single-image models (UniDemoiré), and single-image methods drop to about 40.8 dB on complex figures in LCDMoire. The focused-only baseline quantifies how much the optical reference adds. If that gap is small, the dual-camera premise itself is weak.

Proposed Experiments

The decisive check is a curve of PSNR, SSIM and LPIPS against synthetic disparity for the four fusion variants: our model, flow + concatenation, unaligned concatenation, and deformable convolution. We plot the curve separately for all pixels, for repetitive regions and for occluded regions. Our claim requires a visibly flatter curve in the latter two. The second check examines the gate. Mean $g$ inside occlusion masks and inside in-focus regions should be significantly lower than elsewhere, and the gate-ablated model should lose most of its accuracy in exactly those regions. The third check covers temporal stability (t-MSE, which the source paper reports without a numeric value) and wall-clock runtime at full HD against the FlowFormer pipeline. Finally, we evaluate on DualReal (50 real videos) and RealTextureMoire. If paired ground truth is unavailable there, a question the file does not settle, we fall back to LPIPS against the defocused view in low frequencies and a blind preference study. No reported result was found on Lacuna for any method under controlled stereo disparity, so every point on the main curve will be new measurement.