Forecasting where a pedestrian will walk gets much better when the model can see their body. Knees, ankles and shoulders turn before the person's center does. But the keypoint detectors that supply this body pose fail in the dark, which is exactly when pedestrians are hardest to see. We propose to fix this on the training side rather than at test time. Real night images are not made brighter. Instead, labeled daytime driving video is converted into realistic night video, and the pose-and-trajectory pipeline is trained on that. The conversion is done mostly with physics. Camera pixels are first undone back into linear light, meaning values proportional to photon counts. In that space the scene is darkened, signal-dependent sensor noise is added, and the camera's curve is reapplied. A small learned component trained without paired images adds only what physics cannot write down: street lamps, headlights and glow. That component is additive in linear light, because light from separate sources really does add there.
The direction of translation is the main bet. Turning night into day forces a generator to invent detail the sensor never recorded. Turning day into night only removes information, so the pedestrian's limbs, and their keypoint labels, stay where they were. The non-rigid motion of walking bodies plays a specific role. Translation that flickers from frame to frame produces jittery pose sequences, and jitter is poison to a forecaster that reads pose over time. So we tie the learned residual across frames using the dense, deforming motion field estimated on the well-lit daytime source video. The claim to test is this: a forecaster trained this way keeps most of the accuracy benefit of body pose at night, while forecasters fed by test-time enhancement or naive augmentation lose it.
Where the Pieces Come From and What Is Missing
Three bodies of work motivate the gap. Social-Pose adds a Transformer encoder over skeletal keypoints to existing trajectory predictors. It reports about 25% lower average displacement error and 29% lower final displacement error on the synthetic JTA dataset at 2% extra inference cost, but it assumes clean upstream keypoints and acknowledges degradation in low light. nuScenes measures how bad the dark is: camera-based detection mAP drops 55–58% at night, against 36% for LiDAR. Pedestrian-intent work such as PIE was collected only in daytime, so it gives no night signal at all. Nobody in this material measures whether the pose benefit survives at night, or builds training data to make it survive.
Two lines of work translate night imagery and constrain how we design ours. ToDayGAN translates night driving frames to day without paired images. It uses separate discriminators on blurred color, luminance and image gradients, and lifts RobotCar Seasons localization from 13.9% to 52.9%. It works purely in camera RGB, frame by frame. Nighttime Visibility Enhancement does model the camera curve: a linearization network recovers the inverse response from a single night image before suppressing glare, reaching 23.51 PSNR against 23.01 for SingleHDR on HDR-Real. It also documents why recovering linear light from night images is hard. Its gray-world prior breaks on white glare, and inverting the camera curve is ill-conditioned in deep shadow and at saturation. Our design sidesteps that difficulty by linearizing day images, whose pixels sit mostly in well-conditioned mid-tones.
The non-rigid motion literature supplies the method and a warning. Deformation-Aware Unpaired Image Translation shows that standard CycleGAN deletes or invents thin limbs. On fruit flies, keypoint accuracy (PCK@15px) is 72.9% for CycleGAN versus 84.7% for their model, which separates shape warping from texture painting. Their remedy separates geometry from appearance. We go further and forbid geometry change entirely, which is legal because day and night share the scene. Region Deformer Networks model each moving object's non-rigid motion with a 32-coefficient bicubic field. SplatFlow distills 2D optical flow into continuous motion fields instead of rigid boxes. Both use these fields only to explain past frames. We reuse that kind of dense flow, computed on the daytime source where it is reliable, as the constraint that keeps synthesized night video temporally stable on deforming bodies.
Core Claim
Let $x_t$ be a daytime sRGB frame at time $t$ and $f$ the camera response function, the curve mapping linear irradiance to stored pixel values. We parameterize $f$ with the one-latent-variable model of Single Latent Variable CRF, which fits the 201 curves of the DoRF database with mean RMSE $5.61\times10^{-4}$. The synthesized night frame is
$$\tilde{x}_t = f\big(\mathrm{clip}(\alpha\, f^{-1}(x_t) + r_t + n_t)\big),$$
where $\alpha \ll 1$ is a sampled exposure-and-illumination scale. The term $r_t \ge 0$ is the learned linear-light residual containing lamps, headlights and glow. The term $n_t$ is Poisson–Gaussian sensor noise with variance $a(\alpha f^{-1}(x_t) + r_t) + b$, whose parameters $a$ and $b$ are sampled from a plausible sensor range. The residual generator is trained adversarially against real night frames and regularized to move with the scene:
$$\mathcal{L}_{\text{temp}} = \big\| r_t - W_{t-1\to t}(r_{t-1}) \big\|_1 \ \text{on non-occluded pixels},$$
where $W_{t-1\to t}$ warps by dense optical flow estimated on the daytime frames $x_{t-1}, x_t$. Pedestrian regions use per-instance non-rigid flow. Geometry is never altered, so daytime keypoints, boxes and future trajectories transfer to $\tilde{x}_t$ unchanged.
The pose estimator and a pose-conditioned multi-modal forecaster are then trained on a mix of real day and synthesized night clips. "Multi-modal" here means the forecaster outputs several distinct futures rather than one averaged path; we use a Social-Pose encoder on a Trajectron++-style conditional latent-variable predictor. What would be new is the combination of three things: synthesis in the day-to-night direction, done in linear light, with temporal coupling along non-rigid motion. It would also be the first measurement, to our knowledge, of how much pose actually helps forecasting at night.
Research Program
The work has three stages. The first builds and validates the synthesizer on nuScenes daytime camera sequences, with real nuScenes and RobotCar night frames as the unpaired target set. Before any forecasting result means anything, the synthesizer must reproduce the real degradation it claims to model. To test this, a camera-only detector trained on day data is evaluated on synthesized-night versions of daytime validation scenes. We check whether its mAP drop falls near the 55–58% real day-to-night drop that nuScenes reports for vision methods. A synthesizer that degrades too little would teach nothing, and one that degrades too much would teach the wrong thing.
The second stage trains keypoint estimation and forecasting. Pedestrian trajectories come from nuScenes annotations. Day keypoint pseudo-labels come from a strong estimator run on day frames and are copied to the synthesized night frames, which is legal because geometry is unchanged.
The third stage analyzes where the gains come from. We separate keypoint stability from per-frame keypoint accuracy, and we separate the physics operator from the learned residual. We also examine scenes with white glare, where the linearization network's gray-world prior is known to fail, to see whether synthesis-side linearization avoids that failure.
Baselines
The naive reference is a pose-conditioned forecaster trained only on real daytime data and run directly on real night frames. Next to it sits the same forecaster with pose removed, using only past positions and boxes. The difference between these two, measured at day and at night, is the central quantity of the study: the night pose benefit. For grounding, Social-Pose reports roughly 25%/29% displacement-error gains from pose on JTA. We found no reported result on Lacuna for pose-conditioned or box-only pedestrian forecasting on nuScenes night scenes. We also found no reported Trajectron++ ADE/FDE numbers on nuScenes in the file, so all of these numbers will be produced here under one protocol.
The strongest existing test-time method is ToDayGAN. It translates each real night frame to day, and the unchanged day-trained pose estimator and forecaster then run on the output. It is the strongest competitor because it is the best night-to-day translator in the file for a downstream geometric task, at 52.9% versus 13.9% on RobotCar Seasons. We retrain it on the same nuScenes and RobotCar unpaired sets with the same generator capacity and number of training iterations as our residual generator. It must beat our method on night forecasting error to win.
The cheapest training-side competitor is conventional photometric augmentation of day frames in sRGB: random gamma, brightness, contrast and additive Gaussian noise, with the tuning budget matched to ours. A second training-side baseline is a CycleGAN-style day-to-night generator in sRGB with no physics and no geometry lock.
Three ablations isolate our ingredients. The first is physics-only synthesis with $r_t=0$, which tests whether the unpaired learned part earns its place. The second is our full operator with the curve inversion replaced by identity, so that darkening and noise are applied in sRGB; this isolates linearization. The third is the full operator without $\mathcal{L}_{\text{temp}}$, which isolates the non-rigid temporal coupling.
As an upper reference, the forecaster is trained on real nuScenes night trajectories with pseudo-label pose from the best night-capable estimator available. Beating the baselines means lower night minADE, minFDE and miss rate with $K=6$ samples than every baseline. The error bars come from scene-level bootstrap intervals, which matter because nuScenes has relatively few night scenes. It also means recovering a clearly larger fraction of the daytime pose benefit than sRGB augmentation does.
Why not just…?
Why not just translate night to day at test time, as ToDayGAN does? Going from an information-poor domain to an information-rich one invites invention. The ToDayGAN and deformation-aware papers both document invented structure. CycleGAN's keypoint accuracy on worms (PCK@5px) is 45.3% against 90.3% for a geometry-aware model, because thin structures get deleted or fabricated. A pedestrian's lower leg at night is exactly such a thin structure. ToDayGAN also runs frame by frame, so its outputs can flicker. Still, it is a serious baseline that is run head-to-head. If it matches us on night forecasting error, the direction argument fails and we will say so.
Why not just augment day images with darker gamma and Gaussian noise in sRGB? This is the real threat, because it is nearly free. Our reason to expect it falls short is physical rather than empirical. Real sensor noise grows with the signal and is applied before the camera curve, so in sRGB it is concentrated in shadows in a way flat Gaussian noise is not. Lamps and headlights add light, which is only additive before the curve is applied. The linearization-off ablation and the sRGB-augmentation baseline settle this directly. If they close most of the gap to the full method, the correct conclusion is that linear light does not matter for this task, and the paper reports that.
Why not just drop pose at night and forecast from boxes, or from LiDAR? LiDAR does degrade less at night, 36% versus 55–58% for vision in nuScenes. But many deployed stacks and all of the pedestrian-intent datasets here are camera-based. Social-Pose also shows that center-of-mass models react late to turns. Whether pose helps at all at night is an open empirical question, and the night pose benefit measured above answers it. A finding that even our best night pose adds nothing over boxes would be a useful negative result.
Why not just label real night data? Trajectory labels at night exist in nuScenes, and we use them as the upper reference. Body keypoint labels at night do not exist in the file's datasets, and they are hard for humans to annotate in the dark. Synthesis transfers exact daytime keypoints for free. The upper reference tells us how much of the remaining gap labeling would buy.
Proposed Experiments and Decisive Checks
The primary experiment trains every system on nuScenes daytime training scenes, plus each method's own night data or augmentation. All systems are tested on nuScenes night pedestrians with minADE, minFDE and miss rate at $K=6$, with the same forecaster architecture and the same pose estimator backbone across methods. Daytime test results are reported alongside to confirm that night training does not hurt day accuracy.
A secondary check runs the forecaster trained with each method's night data on PIE and JAAD. These are daytime-only per the PIE paper, so this checks only that day accuracy is not hurt; it says nothing about night performance.
To diagnose the temporal component, we report keypoint jitter, defined as frame-to-frame keypoint acceleration on stationary-limb segments, on real night clips. The non-rigid consistency term should reduce jitter even where per-frame keypoint accuracy is unchanged. The forecasting gain should track jitter more closely than per-frame accuracy.
The synthesizer realism check is the detector mAP-drop comparison against the 55–58% reported by nuScenes. We also report the fitting error of the latent-variable curve model on DoRF against its published $5.61\times10^{-4}$, to confirm the curve prior is reproduced.
Three outcomes would falsify the proposal. The first is sRGB augmentation matching the full method. The second is test-time ToDayGAN matching it. The third is the night pose benefit staying near zero for every method. Any one of these would redirect the work: toward cheap augmentation, toward better test-time translation, or away from pose at night.
References
- Social-Pose: Enhancing Trajectory Prediction With Human Body Pose
- nuScenes: A multimodal dataset for autonomous driving
- PIE: A Large-Scale Dataset and Models for Pedestrian Intention Estimation and Trajectory Prediction
- Night-to-Day Image Translation for Retrieval-based Localization
- Nighttime Visibility Enhancement by Increasing the Dynamic Range and Suppression of Light Effects
- Deformation-Aware Unpaired Image Translation for Pose Estimation on Laboratory Animals
- Region Deformer Networks for Unsupervised Depth Estimation from Unconstrained Monocular Videos
- SplatFlow: Self-Supervised Dynamic Gaussian Splatting in Neural Motion Flow Field for Autonomous Driving
- Representing Camera Response Function by a Single Latent Variable and Fully Connected Neural Network
- Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data