Cameras on robots and vehicles often see the world dark, hazy, or noisy. A growing line of work trains small image enhancers for the machine that consumes the picture rather than for a human viewer. These enhancers are tuned by backpropagating the loss of a downstream object detector, and the output images can look odd to people while improving detection markedly. Many systems, however, do not stop at boxes. They build a scene graph, a list of objects plus labeled relationships between them such as "person holding cup" or "cup on table," and then reason over that graph. Whether a relationship holds often depends on subtle cues between objects, such as contact edges, occlusion boundaries, and shadows. An enhancer tuned only to find object centers has no reason to keep those cues.
We propose training a lightweight enhancer through a frozen scene-graph generator, using a relationship loss that counts only the evidence a statistical prior cannot supply. Each annotated relationship is weighted by how badly a "guess from object labels alone" baseline would predict it. Gradient pressure therefore falls on the relationships that can only be recovered from pixels. The core test is narrow: on degraded images, does this relation-aware enhancer recover more correct relationships than an equally sized enhancer trained on detection loss alone? It must do so at matched detection accuracy, and it must transfer to relation models it was not trained with.
Core Claim
Detection-driven enhancement, as in Image-Adaptive YOLO and GenISP, fixes the first failure of scene graphs under degradation, which is objects never being found. It does not target the second failure. When pixels carry little information, a relation model falls back on dataset co-occurrence statistics. The SGG benchmark study showed that a "Frequency + Overlap" baseline is already competitive on clean images. HSGG documents the same kind of language-driven relational hallucination in vision-language models.
Our claim is that an enhancer can be pushed to restore the visual evidence that separates relations the prior gets wrong. This is possible without destabilizing detection, provided relation gradients are allowed to change only a low-capacity image-processing front end and never the detector or relation weights.
Positioning Against Prior Work
The enhancement side supplies the method. IA-YOLO uses a 165K-parameter network that predicts 15 settings for differentiable classical filters (defog, white balance, gamma, contrast, tone, sharpen), trained only through YOLOv3's loss. GenISP trains a tiny raw-image pipeline under a frozen RetinaNet teacher and finds that human-perception losses hurt detection. TDFusion learns a pixel-wise fusion loss by meta-learning against segmentation and detection networks. U-SAID sharpens a denoiser by minimizing per-pixel class entropy from a frozen classifier. All four supervise with per-object or per-pixel objectives. We reuse the IA-YOLO-style filter front end and the GenISP-style frozen teacher, but replace the objective with a relationship loss.
The scene-graph side motivates the proposal and also constrains it. Newell et al. and Wang et al. show that missed proposals put a hard ceiling on relationship recall. The SGG benchmark shows that most failures come from object proposals. It also delivers the result that most constrains us: coupling detector and relation training hurts. Its decoupled models reach 75.40 Recall@50 on Open Images versus 66.47 for coupled ones. A naive "train everything end to end on degraded data" design is therefore ruled out. Freezing both networks and letting relation gradients reach only about 15 filter parameters per image is our answer to that negative result.
Two other works are adjacent but not the same. PRISM is the only paper in our file that joins restoration and scene graphs, and it runs the opposite direction: it takes a clean graph as a condition for inpainting edited regions. HSGG and LEO improve relation reasoning on clean RGB images and clean 3D scans respectively, with no front end for degradation. To our knowledge, no work in the file sends relational loss back into an enhancer.
Research Program
The pipeline consists of a parameter predictor, a differentiable filter chain, a frozen detector, and a frozen decoupled relation head. We use Neural Motifs or RelDN as the relation head, following the SGG benchmark codebase. Only the parameter predictor is trained. Its loss is
$$ L = L_{\text{det}} + \lambda \sum_{(s,o,r)} \big(1 - p_{\text{freq}}(r \mid c_s, c_o)\big)\, \mathrm{CE}\big(r,\ \hat p(\cdot \mid s,o)\big). $$
In this loss, $L_{\text{det}}$ is the frozen detector's loss. The sum runs over annotated relationships, each with subject $s$, object $o$, and predicate $r$. $p_{\text{freq}}(r \mid c_s, c_o)$ is the training-set frequency of predicate $r$ for the subject and object classes $c_s, c_o$. $\hat p$ is the relation head's predicted predicate distribution for that pair of boxes, and $\lambda$ trades off the two terms. The weight $1 - p_{\text{freq}}$ is the ingredient that matters. Relationships the prior already explains, such as "person wearing shirt," contribute almost nothing. Rare or prior-contradicting ones dominate.
No degraded scene-graph benchmark exists in our file, so we build one. We render VG150 under low light (exposure reduction plus shot and read noise), synthetic fog (the procedure used for VOC_Foggy in IA-YOLO), and additive Gaussian noise (the U-SAID setting). Each degradation has three severities, and the original annotations are kept.
For real-world validity we take a few hundred images each from ExDark and RTTS, which already carry boxes, and hand-annotate VG150 predicates on them. We state plainly that this real test set is small and that its construction is a cost of the project.
The analysis has three parts. First, we decompose any gain into detection recovery versus relation recovery. We do this by evaluating predicate classification with ground-truth boxes on degraded images. This setting removes the detection bottleneck entirely, so it directly measures whether enhanced pixels carry more relational evidence. Second, we measure prior reliance: the agreement between the relation head's predictions and the frequency baseline as severity increases, before and after enhancement. Third, we test transfer. An enhancer trained against one relation head (Motifs) is plugged unchanged in front of RelDN and in front of the training-free HSGG. HSGG is evaluated on a subset because it takes about 61 seconds per image.
Baselines
The naive reference is the frozen detector and relation head applied directly to degraded images, with no enhancement and no fine-tuning. Every other number is reported as recovery of the gap between this and the clean-image score.
The human-centric reference is an off-the-shelf PSNR-trained restorer for each degradation, placed in front of the same frozen models. GenISP reports conventional ISPs at 21.2% mAP versus 24.5% for its task-driven pipeline in cross-sensor RAW-NOD detection. That result leads us to expect this baseline to underperform, but it has never been measured on relations.
The strongest existing method is an IA-YOLO-style enhancer with the identical filter chain and predictor size, trained only on $L_{\text{det}}$ from the same frozen detector. IA-YOLO reports 37.08% mAP on RTTS versus 28.82% for plain YOLOv3, and 40.37% versus 36.42% on ExDark. No reported result was found on Lacuna for any enhancer on scene-graph metrics under degradation. To keep the comparison fair, this baseline uses the same degraded training images, the same number of steps, and the same hyperparameter search budget as our method.
Beating it means a statistically reliable gain, across three seeds, in scene-graph Recall@50 and mean Recall@50 on degraded VG150, and in ground-truth-box predicate classification. The gain must hold at detection mAP within one point of the baseline. That condition shows the improvement comes from relational evidence and not from better boxes.
Two ablations isolate the key ingredient. The first uses an unweighted relation loss, with all weights equal to one; it tests whether prior-debiasing matters or any relation signal suffices. The second is a U-SAID-style entropy objective on the relation head's predicate distribution, which needs no labels.
A fine-tuning reference retrains the detector alone on degraded VG150, keeping the decoupled design. We also report this fine-tuned detector combined with our enhancer.
Why not just…?
Why not just use a detection-driven enhancer, since detection is the bottleneck? This is the most serious threat. If missed objects explain nearly all degraded-image failures, as the SGG benchmark's error analysis suggests for clean images, the detection-only enhancer will capture most of the gain. The ground-truth-box predicate test settles this directly. If relation accuracy under degradation is already near its clean value once boxes are given, there is little relational evidence left to restore, and we will report that the detection-driven approach suffices. The proposal is only worth pursuing if this test shows a residual gap.
Why not just fine-tune the detector and relation head on degraded data? Fully joint fine-tuning is the coupled design that the SGG benchmark found inferior (66.47 versus 75.40 Recall@50 on Open Images). Decoupled detector-only fine-tuning, however, is a real competitor, and it may beat any front end on in-distribution degradations. Our case for the enhancer rests on two properties. It is reusable across relation models without retraining any of them, which the transfer experiment tests. It also stacks with a fine-tuned detector, which the combined run tests. If fine-tuning alone matches the combination and the enhancer fails to transfer, the front-end idea loses.
Why not just let the language prior fill in relations when images are bad? The prior can predict relations only between objects that were detected. Newell et al. show missing nodes cap recall regardless of reasoning. HSGG also documents that prior-driven predicates are frequently wrong ("in" boat versus "beside" boat). Our prior-reliance measurement quantifies this. If agreement with the frequency baseline rises with severity while mean Recall falls, then prior fallback is the failure, not the cure.
Why not just use an unsupervised sharpness or entropy objective like U-SAID? It needs no relation labels, which is attractive. Per-pixel entropy, however, carries no information about which pairwise configuration holds. The entropy ablation runs this idea at the relation level, so a label-free success would be visible and would be reported as a cheaper alternative.
Decisive Checks
The project succeeds if three conditions all hold on synthetic degraded VG150 and on the small annotated ExDark and RTTS sets. First, ground-truth-box predicate classification shows a degradation gap that detection-only enhancement does not close. Second, the prior-weighted relation loss closes a meaningful fraction of that gap at matched detection mAP, outperforming the unweighted ablation mainly on mean Recall over rare predicates. Third, the gain survives transfer to an unseen relation head.
The project fails, and we will report that detection-driven enhancement is enough, if the first condition does not hold. It also fails if adding the relation term lowers detection mAP, which would extend the coupling penalty from the SGG benchmark from detector weights down to pixels.
References
- Image-Adaptive YOLO for Object Detection in Adverse Weather Conditions
- GenISP: Neural ISP for Low-Light Machine Cognition
- Image Scene Graph Generation (SGG) Benchmark
- HSGG: Training-Free Hierarchical Scene Graph Generation with Geometry-Guided Relation Reasoning
- Task-driven Image Fusion with Learnable Fusion Loss
- Segmentation-Aware Image Denoising without Knowing True Segmentation
- Pixels to Graphs by Associative Embedding
- Comprehensive Image Captioning via Scene Graph Decomposition
- PRISM: Progressive Restoration for Scene Graph-Based Image Manipulation
- Edge-Centric Relational Reasoning for 3D Scene Graph Prediction