Direct Machine Vision on Raw Sensor Data

This direction is merged into another direction: linear signal preservation. Its last version stays readable and citable here.

Version 1 of 144 papers62 conceptsfrozen 2026-10-04

Traditional computer vision pipelines are built on a fundamental irony: we capture high-fidelity physical signals from the world, immediately mangle them to satisfy the quirks of the human eye, and then ask neural networks to make sense of the result.

Standard Image Signal Processors (ISPs) perform a series of "beautification" steps—demosaicing, denoising, tone mapping, and gamma correction—to produce 8-bit sRGB images. While these images look good to us, researchers like Gourav Datta and Zeyu Liu argue that this process is a "human-centric tax" that discards critical information and consumes unnecessary power. A growing body of research suggests that machine vision models should instead "see" directly in RAW.

The Information Loss of Aesthetics

The primary reason to bypass the ISP is information preservation. When a sensor captures light, it records a linear radiance signal with high bit-depth (often 12 or 14 bits). The ISP then compresses this into an 8-bit sRGB format. In this transition, the extremes of the signal—the deep shadows and bright highlights—are often clipped or smoothed away.

Zhihao Li and Salman Asif demonstrate that RAW data preserves the full dynamic range of a scene, allowing models to detect objects in extreme lighting where sRGB versions would appear pitch black or blown out. Similarly, Chen Chen and Vladlen Koltun showed in Learning to See in the Dark that processing RAW Bayer patterns directly allows networks to reconstruct clear images from photon-starved environments that traditional denoising filters would simply turn into a grainy mess.

By operating on the raw mosaic, models also avoid "interpolation artifacts." Standard demosaicing guesses the missing color values at each pixel, which can smear high-frequency textures. Federico Lincetto found that training neural rendering models directly on mosaicked data preserves these fine details, leading to sharper 3D reconstructions.

Efficiency Through Omission

Beyond accuracy, the "ISP-less" approach offers a massive efficiency dividend. The ISP is one of the most power-hungry components in a mobile or autonomous system. By eliminating the need to convert light intensity into human-perceivable pixels, researchers have achieved staggering hardware savings.

The $\rho$-Vision framework by Zhihao Li reported a 72% reduction in latency and a 62% reduction in power consumption by performing tasks directly on RAW snapshots. In the video domain, Haichao Wang and Jiangtao Wen integrated motion estimation directly into the Bayer-domain processing, slashing computational costs by over 70% while maintaining near-identical accuracy to RGB-based models.

Bridging the RAW Data Gap

The biggest hurdle to this direction is the "data gap." Most massive datasets (like ImageNet or COCO) consist of sRGB images, not RAW sensor data. To train RAW-native models, researchers have developed several clever workarounds:

  1. ISP Inversion: Gourav Datta and Zeyu Liu proposed "inverting" the ISP to create synthetic RAW datasets from existing RGB labels. Similarly, Radu Berdan introduced ReRAW to bridge the gap between high-quality RGB datasets and the raw data edge devices actually see.
  2. Knowledge Distillation: Alexander Bronstein and Eli Schwartz developed ISP Distillation, where a RAW-based model is trained to mimic the predictions of a pre-trained RGB model, effectively "teaching" the RAW model using the vast knowledge stored in sRGB-trained networks.
  3. Cross-Domain Pre-training: In Towards RAW Object Detection in Diverse Conditions, Zhongyu Li and Xin Jin introduced AODRaw, a strategy that pre-trains vision backbones directly on the RAW domain. This allows the model to interpret high-bit-depth information natively from the start.

Machine-Native Perception

The shift toward RAW processing represents a move from "human-centric" to "machine-native" perception. As Wenjun Huang and Mohsen Imani demonstrate with Dr. RAW, vision backbones can be tuned to be highly efficient and robust by simply respecting the physical properties of the sensor.

When we stop trying to make sensor data "look good" for people, we unlock a version of computer vision that is faster, leaner, and more capable of seeing in conditions that leave humans—and traditional cameras—in the dark.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_direct_machine_vision_on_raw_sensor_data_4e8f1136,
  title = {Direct Machine Vision on Raw Sensor Data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/direct-machine-vision-on-raw-sensor-data/rdr_4e8f1136e7554221840e26457f12a636},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_direct_machine_vision_on_raw_sensor_data_4e8f1136_v1,
  title = {Direct Machine Vision on Raw Sensor Data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/direct-machine-vision-on-raw-sensor-data/rdr_4e8f1136e7554221840e26457f12a636/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}