LC-ISP Benchmark: Learning Advanced ISP with Live and Capture Streams

Kai Chen, Junyu Lou, Jinhua Zhang, Weiyi You, Shuhang GuarXiv 2026 · arXiv

In the world of smartphone photography, what you see on your screen (the "live" preview) is often a pale shadow of the final photo you get after pressing the shutter. This discrepancy exists because your phone’s processor has milliseconds to render the preview, while it can take much longer to fuse multiple frames and apply heavy-duty neural processing for the final "capture."

A new paper from researchers at the University of Electronic Science and Technology of China turns this performance gap into a powerful training signal. By introducing the LC-ISP Benchmark, they’ve created a way to train lightweight, high-quality Image Signal Processors (ISPs) that bring "capture-grade" quality to real-time "live" streams.

TL;DR

The researchers introduce LC-ISP, a massive dataset of 53.9k image groups that leverages the natural quality gap between a smartphone's live preview and its multi-frame fused capture stream. By using the high-quality capture stream to supervise a lightweight, two-stage neural ISP, they achieved a 2 dB PSNR improvement over state-of-the-art baselines within a strict mobile-friendly computational budget.

What Problem Does This Paper Address?

Traditional Image Signal Processing (ISP) converts raw sensor data (RAW) into the pretty pictures we see on our screens (sRGB). Recently, deep learning has revolutionized this field, but these models are only as good as their training data.

Existing datasets face two massive hurdles:

  1. The Misalignment Headache: Most datasets are "cross-device," pairing a smartphone RAW image with a DSLR sRGB image. Even with sophisticated software, perfectly aligning pixels from two different cameras is nearly impossible, leading to "blurry" supervision.
  2. The Dynamic Range Wall: A single RAW frame from a small mobile sensor has physical limits on noise and dynamic range. Training a model on single frames often fails to capture the "magic" of modern computational photography, which relies on fusing multiple exposures.

The authors realized that the perfect "ground truth" already exists inside the phone. By pairing the live stream (single-frame, low latency) with the capture stream (multi-frame, high fidelity) from the same sensor, they eliminate misalignment while providing a high-quality target for the model to emulate.

Methods

The authors propose a Dual-Stage ISP architecture designed to be efficient enough for mobile deployment (targeting a ~2.5 GFLOP budget).

  • Stage 1: Restoration Net: This module takes the noisy, single-frame RAW input and attempts to reconstruct a "clean" intermediate RGB image. Crucially, it is supervised using the multi-frame fused RGB from the capture stream. This teaches the model to "hallucinate" the denoising and HDR benefits of multi-frame fusion from just a single frame.
  • Stage 2: Enhancement Net: This stage takes the restored RGB and applies spatially adaptive transforms (using bilateral grids) to render the final sRGB image. This decouples the difficult task of signal restoration from the artistic task of color and tone mapping.

Datasets

The core contribution is the LC-ISP (Live-and-Capture ISP) Benchmark:

  • Hardware: Collected using the iPhone 16 Pro.
  • Scale: 53,892 spatially aligned image groups.
  • Diversity: Covers day, night, indoor, outdoor, and high-dynamic-range (HDR) scenes.
  • Structure: Each group contains four images: Live RAW, Live sRGB, Multi-frame Fused RGB (intermediate), and Capture sRGB (ground truth).
  • Comparison: Unlike previous benchmarks like ZRR or MAI, LC-ISP provides 12-bit depth and includes challenging nighttime and HDR scenarios that were previously underrepresented.

Key Insights and Ideas

  • The "Same-Device" Advantage: By using the same sensor for both input and target, the researchers achieved pixel-level alignment. This allows the loss functions to focus entirely on image quality rather than compensating for geometric shifts.
  • Modular Supervision: Most neural ISPs are "black boxes" that go straight from RAW to sRGB. By providing an intermediate "Fused RGB" target, the authors allow for stage-wise learning. This makes the training process more stable and the resulting model more interpretable.
  • Information Density: The authors found that their dataset has higher "information density" than older benchmarks. While models trained on older datasets often plateaued quickly, models trained on LC-ISP continued to improve as more data was added, suggesting the dataset captures a more complex manifold of real-world lighting.

Key Results

The results demonstrate that you don't need a massive model to get great results if your supervision is high-quality:

  • Quantitative: Under a 2.5 GFLOP budget (for 448×448 inputs), the Dual-Stage ISP outperformed the strongest single-frame baselines by over 2 dB PSNR.
  • Qualitative: Visual comparisons show significantly better texture preservation, more robust noise suppression in shadows, and more accurate color reproduction in HDR scenes compared to models like PyNet or AWNet.
  • Few-Shot Adaptation: A model pre-trained on LC-ISP was able to adapt to other datasets (like ZRR) much faster and with fewer samples than models trained from scratch, proving that the "priors" learned from the live-capture gap are highly transferable.

Limitations and Future Work

  • Static Scenes: To ensure perfect alignment for the benchmark, the data was collected on tripods under static conditions. While the model performs well, the dataset doesn't explicitly address the "ghosting" artifacts that occur with fast-moving subjects in multi-frame fusion.
  • Device Specificity: While the iPhone 16 Pro is a top-tier sensor, the specific "look" of the capture stream is tied to Apple's ISP tuning. Future work could involve collecting similar data across a wider variety of sensor manufacturers to create a truly "universal" ISP.
  • Real-time Deployment: While the GFLOP count is low, actual on-device latency depends heavily on NPU (Neural Processing Unit) optimization, which remains an area for engineering refinement.

For those interested in the implementation, the code and dataset are available on GitHub.

Summary updated

Abstract

High-quality paired RAW-to-sRGB datasets are essential for data-driven ISP, yet existing datasets often rely on cross-device collection, causing misalignment and failing to faithfully reflect real capture conditions. We therefore introduce LC-ISP, a large-scale benchmark built from the intrinsic gap between a camera's live preview and offline capture streams. LC-ISP contains 53.9k spatially aligned groups, each comprising live RAW/live sRGB and multi-frame fused RGB/capture sRGB, and covers day, night, indoor, outdoor, and HDR scenes. It also provides capture-stream intermediates for modular supervision. We further develop a compact two-stage baseline that uses these intermediates as auxiliary targets. Under a 2.5-GFLOP budget for 448$\times$448 RAW inputs, it outperforms the strongest single-frame baseline by over 2 dB PSNR while improving texture, denoising, and color reproduction. LC-ISP supports realistic benchmarking, the design of supervision strategies, and the development of lightweight ISP models.
BibTeX
@misc{chen2026lc,
  author = {Kai Chen and Junyu Lou and Jinhua Zhang and Weiyi You and Shuhang Gu},
  title = {{LC-ISP Benchmark: Learning Advanced ISP with Live and Capture Streams}},
  year = {2026},
  eprint = {2611.19284},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19284},
}
Figure