Detectors that fuse a color camera with a thermal camera work well on the dataset they were trained on, then degrade when the scenes, sensors, or weather change. Part of the problem is what they learn about when to trust which sensor. That knowledge sits in fusion weights, gates, or searched architectures that absorb quirks of one dataset, and the searched fusion topologies of MFH-NAS are explicitly described as dataset-adaptive rather than transferable. We propose to pull this one decision out of the network and write it down as a short, readable program of sensor-trust rules. An example rule: "when the visible frame is dark and thermal contrast inside a box is high, weight thermal detections up." A language model writes and revises these rules using textual gradients, meaning natural-language critiques of where the current rules fail, generated from grouped error reports.
The bet is specific and falsifiable. We expect rules that are conditioned on physically meaningful scene statistics, and found by critique-driven search, to transfer from one visible–thermal dataset to others better than a numerically trained gate given the same statistics. They also cost almost nothing at run time, unlike uncertainty-based fusion that needs repeated stochastic forward passes. If a small learned gate on the same inputs transfers just as well, the idea fails. The experiments are built so that this comparison is the central one.
Core Claim
Textual gradient optimization, where a language model acts as the optimizer of discrete text or code, has so far been used on things that are text all the way down. It has been applied to prompts in RiOT and PREFER, to agent workflows in FlowBot, and to prompts paired with LoRA weights in Words & Weights. The closest sensor-fusion work, CMFS, does use text: it builds patch-importance maps from fixed CLIP class prompts to favor thermal over noisy visible features. But it never optimizes that text.
The obvious bridge would be to tune CMFS's prompts with textual gradients. We think that bridge is weak. CMFS is differentiable end to end, so learned continuous prompt vectors would almost certainly match or beat language-model-written prompts in-domain, at lower cost.
We instead apply textual optimization where gradients do not help. The target is a discrete, non-differentiable policy over frozen per-sensor detectors. The objective we care about is performance on datasets the optimizer never saw, and a training set gives no gradient signal for that. The claim is that critique-driven rule search produces trust rules that are portable, cheap at run time, and inspectable. We also claim that the critique step itself, not merely the use of a language model, is what produces the portability.
Research Program
The perception stack stays frozen throughout. Following the "knowledge ceiling" argument in Words & Weights, text cannot create detection ability the models lack, so we do not ask it to. We train three detectors on the source dataset from the DAMSDet codebase: a visible-only detector, a thermal-only detector, and the full fused DAMSDet model. DAMSDet uses deformable cross-attention to align physically offset visible and infrared features.
For each frame, and for each candidate box, we compute a fixed set of cheap statistics:
- mean and variance of luminance in the visible image;
- local contrast in the thermal image;
- how well the visible and thermal edges inside the box agree;
- detector confidence and box size from each branch.
The policy is a short Python function, capped in length. It maps these statistics to nonnegative weights over the three detection sets. The weighted sets are then merged with standard weighted box fusion, so the final score of a merged box is the weighted combination of the branch scores.
Optimization runs on the source-dataset validation split. At each step, the current policy is evaluated. Its errors (missed and false detections) are grouped by statistic bins, for example "dark frames, low thermal contrast, small objects." Each group is summarized with counts and a few representative cases.
A critic model reads this report and writes a critique of which rule caused which errors. This is the textual gradient. An editor model then rewrites the policy. We borrow two stabilizers from RiOT's residual tree search. First, the editor keeps proven clauses and grafts in changed ones, to limit the drift where fixing one failure breaks earlier behavior. Second, we keep a small tree of candidate policies rather than a single chain.
The optimizer never sees target datasets. Every search method in the study gets the same fixed budget of policy evaluations and the same statistics. The deliverables are:
- the rule programs themselves;
- their transfer numbers;
- an analysis of which rules survive across independent optimizer runs. Recurring rules would be evidence that the search finds physical regularities rather than noise, a concern raised by the high run-to-run variance reported for Verbalized Machine Learning.
Baselines
The naive reference point is the frozen fused DAMSDet detector used alone, plus a fixed equal-weight merge of the three branches. Any policy that does not beat both on held-out datasets has no value. DAMSDet is also the strongest existing method on these data. It reports 52.9% mAP on M3FD, 49.3% mAP on FLIR, and 69.6% mAP on LLVIP, each trained in-domain (DAMSDet). We found no reported cross-dataset result on Lacuna for DAMSDet or any other method in the file, so we will measure M3FD-trained transfer ourselves. The in-domain numbers serve only as an upper reference. As a second upper reference we will report DAMSDet retrained on each target dataset. Beating that oracle is not expected.
The decisive ablation is a small learned gate. This is a multilayer perceptron (a small fully connected network) that takes exactly the same statistics and outputs the same three weights. It is trained on the source validation split with the same evaluation budget, spent on hyperparameter and early-stopping selection. We also train a variant with synthetic degradations (fog, glare, flattened thermal contrast) as augmentation. If this gate transfers as well as the text-optimized rules, the gain comes from the chosen statistics and not from textual optimization.
To isolate the critique, we run two further searches over the same policy space:
- Scalar search. Bayesian optimization over thresholds and weights of a rule template, receiving only the scalar mAP.
- Direct rewrite. The editor rewrites the policy from raw grouped errors, with no intermediate critique. Verbalized Machine Learning found this direct style generally more effective than explicit textual gradients, while FlowBot and PREFER argue critiques are needed for credit assignment. Our setting tests that disagreement in a new domain.
Finally, we compare against two further references:
- Expert rules. Rules written by a person in one hour.
- Uncertainty weighting. Uncertainty-weighted fusion in the spirit of the thermal–radar system of UTM, which estimates feature uncertainty with repeated dropout passes and reports 0.644 mAP50:95 at 5–7 FPS on a Jetson. We implement it as per-branch score entropy plus dropout variance on the frozen detectors.
We will count the proposal as beating the field if it clears three bars. First, it exceeds the learned gate's mean target-dataset mAP across held-out datasets by more than twice the standard deviation over five seeds. Second, it matches the fused detector in-domain. Third, it adds under one millisecond per frame.
Why not just…?
Why not just tune CMFS-style prompts with continuous prompt learning? For in-domain accuracy this would likely win, because the model is differentiable and a text-writing optimizer adds only noise and cost there. We concede that setting and do not compete in it. Our question is transfer across datasets. Continuous prompt vectors trained on one dataset are exactly the kind of dataset-fitted parameter that the MFH-NAS authors say fails to carry over. Including a continuous-prompt variant of the learned gate would make this concrete. It is lower priority than the gate itself.
Why not just train a small gate on the same statistics? This is the real threat, and the experiment is built around it. A gate trained on the source dataset can latch onto source-specific correlations between statistics and error. A language model editing rules from critiques tends to write monotone, physically phrased conditions, for example "darker means trust thermal more." Our hypothesis is that those conditions extrapolate. If the gate with degradation augmentation matches the rules on FLIR and LLVIP, we will report that the contribution is the statistics, and drop the textual-optimization claim.
Why not just fine-tune the detector on the target data? This requires target labels, which the deployment setting assumes are unavailable. It is the oracle row, not a competitor. We report it to show how much of the transfer gap the rules close.
Why not just use scalar black-box search over rule thresholds? Scalar search only learns that a policy failed, not which clause failed. FlowBot reports reaching peak performance with about 6x fewer API calls than evolutionary search in text workflows, but that evidence comes from text tasks. The matched-budget comparison, with learning curves of target mAP against the number of evaluations, settles whether the advantage holds here.
Proposed Experiments
Main transfer experiment. We optimize on M3FD and test zero-shot on FLIR_align and LLVIP, using the partitions provided in the DAMSDet codebase. VEDAI serves as a harder aerial, different-sensor test. We report COCO-style mAP and [email protected], the drop from source to target, and worst-group mAP across statistic bins. We repeat the whole study with FLIR as the source, to check that the direction of transfer does not decide the outcome.
Degradation stress test. We apply synthetic fog, glare, and thermal flattening to held-out frames, at severities outside the range used in any augmentation.
Stability analysis. Five optimizer seeds per method. We report variance and a clause-level overlap score between the programs found by different seeds.
Latency. Per-frame overhead of each policy, measured on the same GPU.
Decisive checks. The claim survives only if three things hold. The text-optimized rules must beat the augmented learned gate on held-out datasets beyond seed noise. The version with critiques must beat both direct rewriting and scalar search at equal budget. Rules recurring across seeds must account for most of the transfer gain when the other rules are removed. Failing the first check refutes the proposal. Failing only the second leaves a useful but smaller result: language-model rule search helps, but explicit critique does not, which would be consistent with Verbalized Machine Learning.
References
- MFH-NAS:A Hybrid Neural Architecture Search Framework for Multimodal Fusion Object Detection
- RiOT: Efficient Prompt Refinement with Residual Optimization Tree
- PREFER: Prompt Ensemble Learning via Feedback-Reflect-Refine
- FlowBot: Inducing LLM Workflows with Bilevel Optimization and Textual Gradients
- Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptation
- CMFS: CLIP-Guided Modality Interaction for Mitigating Noise in Multi-Modal Image Fusion and Segmentation
- DAMSDet: Dynamic Adaptive Multispectral Detection Transformer with Competitive Query Selection and Adaptive Feature Fusion
- Verbalized Machine Learning: Revisiting Machine Learning with Language Models
- Robust Human Detection under Visual Degradation via Thermal and mmWave Radar Fusion