A camera hidden behind a phone screen sees the world through the screen's grid of pixels and wiring. That grid scatters light the way a diffraction grating does. Every bright lamp, window, or sun in the scene becomes a huge, spiky, rainbow-fringed flare that spreads across the frame. Where the lamp itself is bright enough, the sensor clips, so its true brightness is lost. Current restoration networks treat this as a dense image-to-image problem. They are fed the display's blur pattern and must learn to undo flare everywhere at once. They struggle most in exactly the saturated, high-contrast scenes that matter. Their residual artifacts also fool downstream recognizers: a diffraction streak can be detected as a lamp, or a real lamp can be erased as flare.
We propose to represent a small, specific part of the scene structurally: the light sources. Each bright source becomes an explicit node with a position, an extent, a color, and a brightness that is allowed to exceed the sensor's clipping ceiling. Every flare pixel is then linked to the source that caused it through the known optics of the display. The key physical observation is that the flare's sidelobes are not clipped, and they scale linearly with the source's true brightness. From them we can recover what the saturated core hid, and then subtract the flare analytically instead of hallucinating over it. Once flare is attributed to its parent source, any "object" that is fully explained by a source's sidelobes can be discarded as an artifact. This addresses the "semantic confusion" failure that Flare7K documents but leaves unsolved.
Core Claim
The dominant, non-local part of under-display degradation comes from a handful of bright sources, so it has very few degrees of freedom. Suppose each source is modeled as a node whose appearance is produced by a physics renderer: the source is convolved with the display's point spread function, which is the image a single point of light makes through the screen. Then two gains follow. First, restoration in saturated regions becomes a small, well-posed fit rather than a hallucination. Second, recognition gets an explicit "explained-by" relation between flare and source, and that relation suppresses false detections. Concretely, the model of the observed raw image is
$$ y = \min\Big(c,\; k \ast \big(b + \textstyle\sum_i a_i\, s_i(\cdot - p_i)\big) + h\Big) + n, $$
where $y$ is the linear raw image, $c$ the sensor saturation level, $k$ the display point spread function, $b$ the non-source background radiance, $s_i$ the unit-brightness shape of source $i$ centered at position $p_i$ with brightness $a_i$, $h$ the low-frequency scattering haze, and $n$ sensor noise.
Fix the positions and shapes, and restrict attention to pixels in an annulus around each source that are unsaturated and dominated by flare. On those pixels the model is linear in the brightnesses $a_i$. They can therefore be recovered by non-negative least squares, even when every pixel of the source core is clipped. Two features of the display optics make this tractable: the point spread function has a very large support, beyond 800×800 pixels, and heavy, spiky tails (DISCNet, Feng et al.). In ordinary deconvolution those features are a nuisance. Here they are an asset, because they spread the source's energy over many measurable, unclipped pixels.
Relation to Prior Work
Two lines of under-display restoration supply the method we build on and also define what we must beat. DISCNet and UDC-UNet feed a measured point spread function into a network as a conditioning signal. The network then generates convolution weights on the fly, and UDC-UNet adds a tone-mapped loss for high dynamic range. FSI instead separates periodic diffraction in the Fourier domain. All three treat the image as a dense field. None represents the sources that generate the flare, so none can use unclipped sidelobes to estimate a clipped core. We reuse their point spread function conditioning for the residual background after analytic flare removal. The work on raw-domain processing (Cho, UDC-SIT) constrains our design: the model above is linear only before the nonlinear camera processing pipeline, so we fit it on raw data.
Three works motivate the structured side. SGSFormer uses instance masks from a segmentation model to steer sparse attention during restoration. Its structure is flat and one-directional, and the authors note that the segmenter itself degrades under display artifacts. Our representation is not masks from an external model; it is a generative account of where flare comes from. LRDif shows that geometric anchors (facial landmarks) stay reliable under display blur, but only for single faces. Flare7K annotates light-source and streak masks for lens flare and shows that flare corrupts stereo matching. It names semantic confusion as open and suggests semantic priors without building one.
From the scene-representation literature we borrow the principle of "derendering": inferring discrete entities whose rendering explains the image. In Generative Scene Graph Networks, a tree of objects and parts is inferred and re-rendered by simple alpha compositing. That structure recovers parts with about 98% recall even when fewer than 100 pixels are visible, against about 86% without the hierarchy. That model has no optics and was tested only on synthetic primitives. We keep its logic of explaining away through a renderer, but we replace compositing with wave-optics convolution and restrict the entities to the ones the physics makes identifiable. We deliberately do not claim that general relational scene graphs in the style of Visual Genome or iterative message passing fix display artifacts. Their predicates are dominated by trivial spatial relations, and even on clean images the relationship accuracy is low (about 8.7% in the Visual Genome baselines). We therefore use scene-graph generation as an evaluation of downstream structure, and use only one relational prior (a source attaches to a physical object) as an ablation.
Research Program
The system has three parts.
Source proposal. A lightweight network runs on the raw image. It finds candidate sources: saturated blobs, plus bright points just below the clipping level. For each it outputs a position, a shape (a small parametric blob or a low-resolution mask), and an initial brightness.
Optical fitting. For fixed geometry, the brightnesses are solved by non-negative least squares on the unsaturated annulus pixels, using the panel's measured point spread function. A few Gauss–Newton refinement steps then adjust position and shape. Haze $h$ is fit jointly as a low-order smooth field, following the characterization of scattering as a uniform low-frequency veil by Song et al.. The rendered source flare and haze are subtracted, and clipped cores are replaced by the recovered brightness.
Residual restoration. A small point-spread-function-conditioned network, in the UDC-UNet style, restores the remaining background $b$. After the fitting step, $b$ contains only local blur and noise rather than scene-wide flare.
The source nodes are then passed downstream as an explicit layer. Any detected object whose region's energy is reproduced within noise by a source's rendered sidelobe is flagged as flare-explained, and a detector or scene-graph generator may discard it.
We also study one structural question in its own right: panel transfer. Sethumadhavan et al. and UDC-SIT report that dense networks trained for one panel fail on another. In our model the panel enters only through $k$, so swapping the point spread function should transfer the flare-removal stage without retraining.
A second use of the same structure addresses the gap between simulation and real hardware. Real scenes contain near-point sources such as LEDs and distant lamps, and their flares are direct samples of the true point spread function. Fitting $k$ from these in-the-wild sources on real captures gives a self-calibration signal that simulated pipelines lack.
Baselines
The naive reference is to do nothing. We run the downstream detector and scene-graph generator on the degraded image after standard camera processing, and pair this with a simple clip-and-notch filter. This measures how much display degradation actually harms structured understanding, which no paper in the file reports.
The strongest restoration competitor on high-dynamic-range display images is UDC-UNet (Liu et al.). It placed second in the MIPI 2022 challenge at 47.78 dB PSNR, with 0.16 s per image on an A6000 (Feng et al.). The first-place entry reached 48.48 dB, but the file gives no reproducible method for it. UDC-UNet is the right target because it already conditions on the point spread function and is explicitly designed for saturation. If any dense model can learn source brightness implicitly, it is this one. We also include DISCNet (43.27 dB on its own benchmark in Feng et al.), FSI (Liu et al.), and SGSFormer (39.10 dB PSNR on T-OLED, Xue et al.) as the structure-guided competitor.
All restorers are retrained on identical raw training data with identical point spread functions, the same training-GPU budget, and the same hyperparameter search budget. Our pipeline is held to a parameter and latency budget no larger than UDC-UNet's.
For downstream structure, the competitor is restore-then-parse: each restorer's output is fed to the same frozen detector and message-passing scene-graph generator (Xu et al., whose clean-image predicate-classification Recall@50 is 44.75%). This is compared against a detector and a scene-graph generator fine-tuned directly on degraded images with the same data.
Three ablations isolate the key ingredient. The first removes the explicit source fitting, so one network does everything at matched capacity. The second gives the system oracle source brightness, which bounds what fitting can buy. The third removes the flare-explained pruning of detections.
Beating UDC-UNet means two things. On full-image PSNR, it means matching within noise. On the metrics that the claim concerns, it means a clear win: error inside flare halos and saturated cores, and error in the logarithm of source brightness. Downstream, the target is fewer flare-induced false detections at equal or better recall of real lamps and windows.
Why not just…?
Why not just restore with UDC-UNet and then run a scene-graph model? This is the default, and it may be good enough on mild scenes. The evidence against it concerns saturation and the real-world domain. Feng et al. note that clipped regions cannot be inverted and must be hallucinated. UDC-SIT shows restorers falling from about 52 dB on synthetic data to 26–28 dB on real captures, which leaves large residual flare. Our experiment settles this by stratifying results by the fraction of clipped pixels. If restore-then-parse matches us in the heavily saturated stratum, the explicit-source idea is unnecessary, and we will say so.
Why not just fine-tune the detector on images degraded with the simulated point spread function? This is a real threat on in-distribution synthetic data, and we expect it to do well there. Its weaknesses should appear on held-out panels and on real hardware. There, UDC-SIT shows simulation-trained models failing, and Flare7K shows direct training still confuses flare with lamps. The panel-swap test and the real-capture consistency test below decide it.
Why not just protect light-source masks, Flare7K-style, or condition on segmentation masks like SGSFormer? Masks mark where a source is, but not how bright it truly is or which distant streaks it caused. Protecting a source region neither removes its flare elsewhere nor recovers its clipped brightness. The oracle-brightness and no-fitting ablations measure exactly the information that masks lack.
Why not just notch-filter the periodic diffraction peaks? FSI shows that diffraction peaks overlap scene textures in frequency, so static notches remove detail. Notching also cannot touch the non-periodic haze or the saturated cores. We include it only as a naive reference.
Proposed Experiments
Restoration on high-dynamic-range data. We use the MIPI 2022 data, which is simulated with measured point spread functions and has high dynamic range. We report full-image PSNR, SSIM, and LPIPS, together with halo-region and clipped-core error and source-brightness log error, using the simulator's ground-truth radiance.
Panel transfer. We train on P-OLED data and test on T-OLED data, and the reverse, by swapping only $k$. The dense baselines are allowed to swap their conditioning input. The decisive check is whether our flare-removal error on the unseen panel stays near its in-panel value while the dense models degrade.
Downstream structure on Visual Genome. We build an under-display version of Visual Genome. High-dynamic-range sources are inserted at annotated lamps, windows, and screens, and the images are degraded with measured point spread functions, clipping, and noise. We measure three things: the rate of false detections lying inside rendered flare, the recall of real light-emitting objects, and scene-graph Recall@50 and Recall@100 relative to the clean-image model. The decisive result is fewer flare-induced false positives with no loss of real-lamp recall, compared with the best restore-then-parse and fine-tuned baselines.
Real hardware. Using the aligned real pairs in UDC-SIT, we measure PSNR against the reported 26–28 dB range. Because UDC-SIT has no object labels, we measure downstream quality as detection agreement with the same detector's output on the aligned clean capture.
Self-calibration. We test whether fitting $k$ from in-the-wild point sources closes part of the gap between simulation and real hardware. If source-brightness estimation fails on real captures because unmodeled scattering breaks linearity, that negative result bounds the approach, and we will report it.
References
- Flare7K: A Phenomenological Nighttime Flare Removal Dataset
- Removing Diffraction Image Artifacts in Under-Display Camera via Dynamic Skip Connection Network
- UDC-UNet: Under-Display Camera Image Restoration via U-shape Dynamic Network
- FSI: Frequency and Spatial Interactive Learning for Image Restoration in Under-Display Cameras
- Panel-Specific Degradation Representation for Raw Under-Display Camera Image Restoration
- UDC-SIT: A Real-World Dataset for Under-Display Cameras
- Segmentation Guided Sparse Transformer for Under-Display Camera Image Restoration
- Lrdif: Diffusion Models For Under-Display Camera Emotion Recognition
- Generative Scene Graph Networks
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
- Scene Graph Generation by Iterative Message Passing
- Under-Display Camera Image Restoration with Scattering Effect
- Transform Domain Pyramidal Dilated Convolution Networks for Restoration of Under Display Camera Images
- MIPI 2022 Challenge on Under-Display Camera Image Restoration: Methods and Results