Particle physicists turn blurry detector readings into sharp measurements with a tool called kinematic constraint fitting. Every measured particle comes with a known error bar. A least-squares fit nudges each measurement toward values that satisfy exact conservation laws, and it moves the uncertain measurements more than the precise ones. The size of the nudge needed, summarized as a $\chi^2$ score, then tells you whether the event is real or a random combination of unrelated tracks. This only works because the error bars are trustworthy. Camera-based tracking has the same structure: noisy per-frame detections, physical constraints on how objects move, and many false pairings between detections. Yet it almost never gets trustworthy error bars, because the camera's image signal processor (ISP), the on-chip pipeline that turns sensor counts into a pretty 8-bit picture, discards exactly the information needed to compute them. Its tone curves, denoising and clipping erase the link between pixel value and photon count.
This note argues that the most useful thing RAW sensor data offers temporal vision is not only better per-frame features but a physically derivable error model. In linear RAW data, pixel noise follows a known signal-dependent law set by the sensor's gain and read noise. Saturated pixels are known to be saturated. That makes it possible, for the first time in camera tracking, to run a genuine constrained $\chi^2$ fit whose scores are calibrated. Calibrated scores support rejection of false detections and wrong associations at a controlled false-alarm rate, especially at night and under glare, where the ISP throws the most away.
Core Claim
The claim is that detections computed from RAW frames can carry covariances (error ellipses on box position and size) that stay statistically calibrated across illumination levels and across sensors when those covariances are informed by the sensor noise law in the linear domain. Calibrated covariances turn a standard multi-frame constrained fit into a working background-rejection filter in the sense of high-energy physics. The same machinery on post-ISP sRGB frames, even with a learned uncertainty head, will give miscalibrated scores precisely in the regimes where tracking is hardest. The measurable signature is the "pull," the fit residual divided by its predicted error. With RAW-informed covariances it should stay near a unit-width Gaussian from bright day to deep night. With sRGB it should widen and develop tails as light drops, which degrades $\chi^2$-gated association.
This also resolves a tension the RAW detection literature leaves open. Raw or Cooked? Object Detection on RAW Images shows that naive RAW input collapses detection to 31.3 AP versus 50.5 AP for ISP-processed RGB on PASCALRAW. A learnable Yeo-Johnson power transform, which reshapes skewed sensor values toward a Gaussian, recovers 52.6 AP. Toward RAW Object Detection: A New Benchmark and A New Model names the same failure on 24-bit data the "RAW Paradox." Those nonlinear transforms are what a neural network needs, but they destroy the linearity a $\chi^2$ fit needs. Our design separates the two jobs. The network sees any transform it likes to compute features, and the error model is computed from the untouched linear data and passed alongside.
Research Program
The pipeline has three parts. The first is a RAW detector in the style of Dr. RAW, which processes Bayer data through a light sensor-mapping matrix and adapters. To it we add a covariance head that predicts a full $4\times4$ covariance for each box's center and log-size. The head's extra input is a variance map computed from the linear RAW frame with the standard photon-plus-read noise law:
$$\mathrm{Var}[x] = a\,x + b,$$
where $x$ is the linear pixel value after black-level subtraction, $a$ is the sensor gain (which grows with ISO) and $b$ is the read-noise variance. Both come from metadata or a one-time calibration. Saturated pixels are marked with infinite variance. RAW makes this possible because the sensor knows which pixels clipped, whereas in sRGB a value of 255 could be either clipped or genuinely white, which is the ambiguity Exploiting Raw Images for Real-Scene Super-Resolution and Invertible Image Rescaling describe as irreversible tone and quantization loss. The head is trained with a Gaussian negative log-likelihood on box errors, so that its outputs are meant to be variances rather than confidence scores.
The second part is a sliding-window constrained fit over $K$ frames for each candidate track. Each frame contributes a measurement $m_t$ (box center and log-size) with predicted covariance $\Sigma_t$. A trajectory model $f_t(\theta)$, here constant acceleration in the image plane over short windows, is fit by minimizing
$$\chi^2(\theta) = \sum_{t=1}^{K} \big(m_t - f_t(\theta)\big)^\top \Sigma_t^{-1} \big(m_t - f_t(\theta)\big) \quad \text{subject to} \quad g(\theta) = 0,$$
where $\theta$ holds the trajectory parameters and $g$ collects hard constraints solved with Lagrange multipliers. This is the formulation used in the 4C fits of Partial wave analysis of J/ψ→γ(K+K−π+π−), which cut $7.8\times10^6$ triggers to 1516 clean events. Our constraints play the role of the extra mass constraints in the 5C and 6C fits of Study of... and Measurements of the branching fractions of ψ(3686)→Σ¯0Λ+c.c. and χcJ→ΛΛ¯. They are a shared aspect ratio for a rigid object across the window, the analogue of a fixed resonance mass. In controlled captures they also include a known camera or target motion read from an encoder, the analogue of the precisely known beam energy that Absolute branching fractions of Cabibbo-suppressed D→KK¯ decays substitutes into the beam-constrained mass.
If the covariances are correct, the minimized $\chi^2$ follows a chi-square distribution with degrees of freedom equal to measurements minus free parameters plus constraints. The fit therefore returns a p-value for every association hypothesis. A competing pairing is resolved by comparing $\chi^2$ across hypotheses. Occlusion or a sudden maneuver shows up as a high $\chi^2$, much as an unobserved particle breaks a 4C fit. That case triggers a fallback to fewer constraints rather than a distorted forced fit.
The third part is a diagnostic suite borrowed directly from physics practice. It contains pull histograms, $\chi^2$ probability distributions (which should be flat for true tracks), and their dependence on illuminance, sensor and object size. Physicists use the shape of these distributions to detect detector miscalibration. We use them to measure how much error-model information the ISP has destroyed.
The positioning against adjacent work is as follows. RadProPoser is the closest in spirit: it estimates human pose with uncertainty from raw radar. It stops at per-frame uncertainty and never imposes constraints or tests calibration through a goodness-of-fit statistic. WristCompass learns kinematic coupling as a soft visual cue on RGB video, whereas we impose explicit constraints and need trustworthy covariances to do so. How Well do Diffusion Policies Learn Kinematic Constraint Manifolds? asks whether learned generators respect constraints without explicit projection. That question constrains our design, because it is why we enforce constraints in a solver rather than hoping a network learns them. The RAW detection works above all operate on single frames, and SFU-HW-Tracks-v1 provides tracking labels only on uncompressed YUV video, not on Bayer data.
Baselines
The naive reference point is a standard tracker that pairs detections frame to frame by box overlap with no error model. It runs once on sRGB detections from an ISP-processed image and once on RAW detections. Every gain we claim must be measured against it.
The strongest competitor on the actual question is a post-hoc kinematic fit on sRGB. This is the same windowed constrained fit, run on detections from the same detector architecture trained on ISP-processed frames, with a covariance head trained by the same negative log-likelihood. It is the most dangerous baseline because it differs from our system only in whether the error model can see linear photon statistics. To keep the comparison fair, both arms share the architecture, the pretrained initialization, the training images (the sRGB arm receives the same frames rendered through a fixed ISP), the number of training steps and the hyperparameter search budget. We tune the $\chi^2$ gate threshold separately for each arm on a held-out split.
A stronger variant of this baseline feeds the sRGB covariance head an analytic noise level computed from exposure metadata (ISO and exposure time). This tests whether the benefit comes from RAW itself or merely from knowing the camera settings.
For the front-end detector, the reference is Dr. RAW. It reports 72.1 mAP on the LOD low-light detection dataset, 4.9 points above the prior best, and 88.3+ mAP in zero-shot transfer from Nikon to iPhone and Samsung sensors. We reimplement its adapter-based RAW pipeline as our detector so that tracking gains cannot be attributed to a better per-frame detector. For tracking itself on RAW video there is no reported result found on Lacuna, and none for SFU-HW-Tracks-v1 tracking accuracy either. We therefore establish all tracking numbers ourselves under one protocol.
Two ablations isolate the key ingredient. The first removes the linear variance map from the RAW covariance head, leaving a learned-only uncertainty. The second keeps the map but replaces the trained covariance with the analytic variance propagated through a fixed box-localization sensitivity, leaving a physics-only uncertainty. Beating the strong baseline means two things at matched detection recall. Pull widths must stay within a fixed tolerance of one across illumination bins where the sRGB arm drifts. And the RAW arm must show a lower false-track rate and lower trajectory error in the darkest third of the data, with the gap shrinking toward zero in bright scenes. A uniform gain at all light levels would point to a better detector rather than a better error model.
Why not just…?
Why not just run a Kalman filter on sRGB detections? A Kalman filter is the recursive form of the same least-squares fit, so it is no simpler in principle, and it needs the same measurement covariance. The evidence that sRGB undermines that covariance is indirect but consistent. In Towards RAW Object Detection in Diverse Conditions, sRGB-trained models lose about 13% under maximum sensor noise versus about 4% for RAW-pretrained ones. Dr. RAW reports dark segmentation collapsing to 5.02 mIoU through a standard ISP against 38.46 mIoU from RAW. Whether this per-frame fragility actually miscalibrates track covariances is exactly what the pull-versus-illuminance experiment measures. If sRGB pulls stay near unit width, this proposal's main claim is wrong.
Why not just learn the uncertainty end-to-end on sRGB, perhaps with ISO metadata? This is the real threat. A network can infer noise level from image texture, and an ISO-derived prior may give most of what the RAW variance map gives. Our counter-hypothesis is that ISP denoising removes the texture the network would need, and that adaptive tone mapping makes ISO a poor proxy for per-region photon count. Per-region photon count also varies within a frame, as with headlights against dark road. The metadata-augmented sRGB baseline exists to settle this. If it matches the RAW arm, we will report the narrower finding that exposure metadata, not RAW, is what calibrated tracking needs.
Why not just use a better RAW detector and an ordinary tracker? Better per-frame detection helps, and Dr. RAW-style pipelines already deliver it. But an ordinary tracker cannot say how unlikely a given association is, so it cannot reject false tracks at a controlled rate or detect when its own motion model breaks. The naive-tracker-on-RAW arm measures how much of the gain is per-frame detection alone. The constrained fit has to add measurable value beyond it, particularly in false-track rate, for the proposal to justify itself.
Proposed Experiments
Data comes from three sources of increasing realism. The first is semi-synthetic sequences built from real RAW stills in PASCALRAW (Nikon D3200, 12-bit), AODRaw (7,785 images, 62 classes, rain, fog and low light) and ROD (Sony IMX490, 24-bit, day and night driving). We apply known camera homography trajectories in the linear domain, re-mosaic, and redraw sensor noise independently per frame at photon levels scaled to simulate illuminance from daylight to near darkness. Because the exact kinematics are known, this plays the role that Monte Carlo simulation plays in the physics analyses. Rendering the same frames through a fixed ISP gives the paired sRGB arm. The second source is dynamic-scene RAW video from Supervised Raw Video Denoising With a Benchmark Dataset on Dynamic Scenes, RViDeformer and Real-RawVSR, on which we hand-annotate tracks for a subset of sequences. The third is a small new capture of targets on a motorized rail with an encoder, recorded at several lux levels on two sensors. It gives real noise together with a known-motion hard constraint.
The metrics are pull width and tail fraction, the uniformity of $\chi^2$ p-values for true tracks, false-track rate at fixed recall, identity-switch counts and IDF1, and trajectory RMSE against ground-truth kinematics. All are reported per illuminance bin and per sensor. Cross-sensor tests train on Nikon and evaluate on Sony and the rail-capture sensors, with only the noise parameters $a$ and $b$ updated from metadata. This tests the prediction that a physics-informed error model transfers where a learned-only one does not.
Three outcomes would be decisive. The first is the pull-width curve against illuminance for RAW, sRGB and sRGB-with-metadata covariances. Our claim requires the RAW curve to stay flat while the others rise in the dark. The second is the false-track rate at matched recall in the darkest bin, where we expect the RAW constrained fit to beat both the sRGB fit and the RAW detector with an ordinary tracker. The third is a gain that disappears in bright scenes, which separates an error-model effect from a detector effect. If the metadata-augmented sRGB arm closes the gap on all three, the research answer is still useful, but it is that knowing the camera settings, not bypassing the ISP, is what makes physics-style fitting work for cameras.
References
- Raw or Cooked? Object Detection on RAW Images
- Toward RAW Object Detection: A New Benchmark and A New Model
- Dr. RAW: Towards General High-Level Vision from RAW with Efficient Task Conditioning
- Exploiting Raw Images for Real-Scene Super-Resolution
- Invertible Image Rescaling
- Partial wave analysis of J/ψ→γ(K+K−π+π−)
- Study of<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:msup><mml:mrow><mml:mi mathvariant="normal">e</mml:mi></mml:mrow><mml:mrow><mml:mo>+</mml:mo></mml:mrow></mml:msup><mml:msup><mml:mrow><mml:mi mathvariant="normal">e</mml:mi></mml:mrow><mml:mrow><mml:mo>−</mml:mo></mml:mrow></mml:msup><mml:mo stretchy="false">→</mml:mo><mml:mi mathvariant="normal">p</mml:mi><mml:mover accent="true"><mml:mrow><mml:mi mathvariant="normal">p</mml:mi></mml:mrow><mml:mrow><mml:mo accent="true" stretchy="false">¯</mml:mo></mml:mrow></mml:mover><mml:msup><mml:mrow><mml:mi>π</mml:mi></mml:mrow><mml:mrow><mml:mn>0</mml:mn></mml:mrow></mml:msup></mml:mrow></mml:math>in the vicinity of the<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:mi>ψ</mml:mi><mml:mo stretchy="false">(</mml:mo><mml:mn>3770</mml:mn><mml:mo stretchy="false">)</mml:mo></mml:mrow></mml:math>
- Measurements of the branching fractions of ψ(3686)→Σ¯0Λ+c.c. and χcJ(J=0,1,2)→ΛΛ¯
- Absolute branching fractions of Cabibbo-suppressed<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mi>D</mml:mi><mml:mo>→</mml:mo><mml:mi>K</mml:mi><mml:mover accent="true"><mml:mi>K</mml:mi><mml:mo>¯</mml:mo></mml:mover></mml:math>decays
- RadProPoser: A Framework for Human Pose Estimation with Uncertainty Quantification from Raw Radar Data
- WristCompass: Kinematic Coupling as a Learnable Visual Concept for Ego-Camera Orientation
- How Well do Diffusion Policies Learn Kinematic Constraint Manifolds?
- SFU-HW-Tracks-v1: Object Tracking Dataset on Raw Video Sequences
- Towards RAW Object Detection in Diverse Conditions
- Supervised Raw Video Denoising With a Benchmark Dataset on Dynamic Scenes
- RViDeformer: Efficient Raw Video Denoising Transformer With a Larger Benchmark Dataset
- Real-RawVSR: Real-World Raw Video Super-Resolution with a Benchmark Dataset