Chest X-ray datasets are dominated by common findings, while dangerous but rare conditions may appear only a handful of times. Generating extra training images for those rare conditions is an obvious remedy. Current image generators, however, invent anatomy: they add pacemakers nobody asked for, warp the heart outline, or bend ribs while painting in a lesion. The recent fix is to fine-tune the generator with reinforcement learning against several automatic "judges," one for anatomy, one for whether the disease is visible, and one for text–image agreement. These judge scores are then added together into a single number.
This proposal argues that adding the scores together fails specifically where augmentation matters most. The judge that checks disease presence is itself a classifier trained on the same skewed data, so it is least reliable on the rarest classes. With one summed score, the generator can buy reward on a rare class by exploiting that weak judge, and it pays for this with anatomical damage the sum lets it afford. We propose to keep anatomy out of the sum. Anatomical preservation becomes a hard constraint, enforced through a separate penalty strength for each disease class that rises automatically where the disease judge is being gamed. We then measure success by gains on rare classes in long-tailed chest X-ray benchmarks, not by image-realism scores.
Core Claim
The damage from collapsing several objectives into one scalar reward is not uniform. It grows as class frequency falls, because the reliability of the pathology reward falls with it. A generator fine-tuned under a constrained objective will produce rare-disease images that keep the source patient's anatomy intact, and those images will raise tail-class accuracy of a downstream classifier beyond what rebalanced classifiers on real data achieve. The constrained objective has two parts: pathology reward is maximized, anatomical deviation must stay below a threshold, and a separate multiplier is kept per class. A generator tuned under a linearly summed reward, even one with well-tuned weights, will show a reward-hacking gap that widens toward the tail.
Concretely, for a source image $x_0$ and target rare class $k$, the generator $\pi_\theta$ produces an edited image $x$. We solve
$$\max_\theta \; \mathbb{E}_{x \sim \pi_\theta(\cdot \mid x_0, k)}\big[\,r_k(x)\,\big] \quad \text{s.t.} \quad \mathbb{E}\big[\,c_k(x, x_0)\,\big] \le \epsilon_k \;\; \text{for every class } k,$$
where $r_k$ is a pathology-presence reward for class $k$ and $c_k$ is an anatomical-deviation cost: one minus the Dice overlap between segmentations of $x$ and $x_0$, taken over structures that class $k$ should not alter, plus a penalty for newly detected devices. The threshold is $\epsilon_k$. We optimize the Lagrangian $\mathbb{E}[r_k] - \lambda_k(\mathbb{E}[c_k] - \epsilon_k)$ with one multiplier $\lambda_k$ per class, updated by dual ascent.
The rarity-aware ingredient is that $r_k$ is replaced by a pessimistic estimate: the mean of an ensemble of pathology classifiers minus a multiple of their disagreement. This follows the uncertainty gating in GARDO, which used ensemble disagreement to trigger regularization in general text-to-image fine-tuning. Here the disagreement shrinks the reward itself, and it will be largest on tail classes.
Relation to Prior Work
Three papers motivate the proposal by showing that decomposed rewards reduce medical hallucination. CXRL fine-tunes a chest X-ray diffusion model with posture, diagnosis, and CLIP-consistency rewards, but sums them with fixed weights and evaluates only average performance on MIMIC-CXR-JPG. Multi-scale reward learning for 3D MRI likewise sums slice-level and volume-level rewards, and its benchmarks are balanced. MAGIC scores dermatology images on a five-item clinical checklist and edits real images rather than generating from scratch. It then collapses the checklist into chosen-versus-rejected pairs for preference optimization, so it loses which criterion failed. We borrow MAGIC's editing of real source images because it gives us a reference anatomy to constrain against. We differ from all three in refusing to sum anatomy into the reward.
The constraint machinery comes from Safe RLHF. That work separated a helpfulness reward from a zero-threshold cost model for language models and enforced the cost with an adaptive Lagrange multiplier, cutting harmful responses from 53.08% to 2.45%. It never touched images or class imbalance. Our per-class multipliers are the extension that matters here. The theoretical case that a single scalar cannot express "anatomy first, then pathology" priorities is made in Scalar reward is not enough. Checklists Are Better Than Reward Models supplies empirical evidence that monolithic reward models reward superficial traits.
The closest competitor on the long-tail side is Causal Disentanglement for Robust Long-tail Medical Image Generation. It separates anatomy and pathology features with a VAE and swaps pathology tokens across image pairs. It then optimizes the initial diffusion noise at test time toward rare-class exemplars, at a cost of 2–3 minutes per image. It keeps anatomy fixed through its architecture and has no learned reward. We instead keep anatomy fixed through an explicit, measurable constraint and amortize the cost into training, so sampling is cheap enough to produce thousands of tail images.
Two papers constrain the design. Holste et al. show that MixUp hurts on chest X-rays because blending erases thin, high-contrast signs such as the air line of pneumoperitoneum. That rules out interpolation-style augmentation and argues for editing that preserves local detail. Rethinking Perceptual Metrics for Medical Image Translation shows that Fréchet Inception Distance (FID), a realism score computed from ImageNet features, has near-zero correlation with anatomical segmentation accuracy. FID is therefore reported but never used to choose between methods.
Research Program
The work proceeds in three parts.
Measuring the problem. First we establish the phenomenon without any new method. We build a CXRL-style generator with a linearly summed reward and fine-tune it to add each of the 20 NIH-CXR-LT findings to real source radiographs, mostly normal or head-class images. For every class we measure the hacking gap: the difference between the reward classifier's confidence on generated images and the confidence of an independent held-out evaluator. The evaluator is a separately trained classifier plus a radiologist-read subsample. The prediction is that this gap, and the anatomical deviation, grow as training frequency falls.
Building the constrained generator. Second, we build the constrained generator. The cost $c_k$ uses an off-the-shelf chest anatomy segmenter for lungs, heart, clavicles, and mediastinum, together with a device detector. For each class, the invariant structures are those the finding should not alter; for example, lung borders are allowed to change for pleural effusion but the cardiac silhouette is not. These choices are fixed a priori with a radiologist and are not tuned. The pathology reward is the pessimistic ensemble estimate. Per-class multipliers are updated during policy-gradient fine-tuning of the diffusion denoiser.
Downstream use. Third, we use the generated images to augment classifier training. The downstream recipe is the one Holste et al. found strongest: learn features on the natural long tail, then retrain the classifier head with balanced sampling. The synthetic images enter only at the head-retraining stage and optionally in the backbone stage.
A circularity guard runs through the whole program. The ensemble used as reward, the evaluator used to measure hacking, and the downstream classifier are trained on disjoint patient splits. No classifier ever scores images it helped shape.
Baselines
Real data only. The naive reference is empirical risk minimization on real data with softmax cross-entropy. Holste et al. report 1.7% tail accuracy and 0.164 group-averaged accuracy on NIH-CXR-LT for this setting (Long-Tailed Classification of Thorax Diseases).
The strongest real-data method is classifier re-training with balanced sampling (cRT) from the same paper. It reports 30.0% tail accuracy and 0.369 group-averaged accuracy on NIH-CXR-LT, and 36.7% tail accuracy on MIMIC-CXR-LT. We rerun cRT with the same backbone, resolution, and head-tuning budget as every augmented variant, so that any gain comes only from the synthetic images.
For the multi-label CXR-LT setting, the reference point is the challenge's top team, which reached macro mean average precision (mAP) of 0.372 with weighted asymmetric loss (CXR-LT overview). We will not claim to beat that ensemble outright. Instead we add our synthetic images to a single reproduced weighted-asymmetric-loss model and report the change.
Strongest generative competitors. The strongest generative competitor is a CXRL-style triple reward combined linearly (CXRL). CXRL reports average AUROC 0.81 and FID 1.51 on MIMIC-CXR-JPG. We found no reported result on Lacuna for CXRL on NIH-CXR-LT or CXR-LT tail classes. We reimplement it on the same base diffusion model, source images, and number of fine-tuning steps as our method. Its summed-reward weights get a grid search whose size equals the threshold-tuning budget we spend on $\epsilon_k$, so neither method gets extra tuning.
The second generative competitor is Causal Disentanglement (Nie et al.). It reports 86% pathological classification accuracy and an anatomy preservation score of 4.21/5 on MIMIC-CXR. We found no reported tail-benchmark result on Lacuna for it. Because of its per-sample cost, we compare it at both an equal number of images and an equal GPU-hour budget.
Ablations. Three ablations isolate the key ingredients. The first uses a single shared multiplier instead of per-class multipliers, which tests whether rarity-awareness matters. The second removes the ensemble-disagreement pessimism, which tests whether constraint alone suffices. The third places anatomy in the reward as a heavily weighted term rather than a constraint.
What counts as beating the baselines. Beating means two things together. Tail accuracy on NIH-CXR-LT must exceed reproduced cRT, against the reported 30.0%, and must also exceed the linear-reward generator, with non-overlapping confidence intervals across three seeds. At the same time, mean anatomical Dice on invariant structures must not be worse than the linear-reward generator.
Why not just…?
Why not just rebalance the classifier on real data? cRT is cheap and strong, and it is the number to beat. Holste et al.'s own analysis gives the reason it plateaus: when a class has seven training images, the frozen backbone has never seen that disease vary across patients, and reweighting the last layer cannot supply that variation. If our augmented cRT does not beat plain cRT on the tail group, the generative route is not worth its cost, and we will say so.
Why not just tune the weights of the summed reward more carefully? This is the most direct threat. If a well-searched set of linear weights matches the constrained method, the constraint adds nothing. We give the linear baseline an equal tuning budget and allow it per-class weights. The decisive evidence is the hacking-gap curve. If linear scalarization, even when tuned, shows a gap that widens toward rare classes and the constrained method flattens it, the constraint is doing work that weights cannot. If both curves are flat, we drop the claim.
Why not just generate many images from a base model and keep only those that pass the anatomy check? Best-of-N filtering needs no reinforcement learning and enforces the same constraint, so it is a serious alternative. Its weaknesses are acceptance rate and selection pressure. The CXRL and MAGIC evidence suggests base models rarely satisfy clinical criteria; MAGIC found only 2.0% of plain text-to-image outputs met three or more criteria. Filtering on pathology score also selects for whatever fools the weak tail classifier. We run filtering at matched compute, using the same total generator forward passes, with both the anatomy constraint and the pessimistic pathology score. If filtering matches our tail accuracy, we will recommend filtering and report the constrained fine-tuning as unnecessary.
Why not just use MixUp or loss reweighting? Holste et al. directly report that MixUp degrades chest X-ray classification. Loss reweighting is already part of our strongest baseline: we add synthetic data on top of it rather than replacing it.
Proposed Experiments
Training and generation use NIH-CXR-LT, with 20 classes and the rarest at 7 images, and MIMIC-CXR-LT, with 19 classes. Multi-label evaluation uses the CXR-LT challenge data with 26 findings. Text-mined labels are noisy; the challenge reports an average 12.3% drop in average precision on its 406-image expert-annotated gold set. We therefore report tail-class AP on that gold set as the headline multi-label metric.
The main metrics are as follows:
- head, medium, and tail accuracy and their group average on NIH-CXR-LT and MIMIC-CXR-LT;
- macro and tail mAP on CXR-LT, including the gold set;
- Dice on invariant anatomical structures and the rate of newly appearing devices;
- the per-class hacking gap;
- a blinded radiologist rating of a stratified subsample for anatomical plausibility and presence of the finding.
FID is reported only for comparability with CXRL and Causal Disentanglement.
There are three decisive checks. The first is a regression of hacking gap and anatomical deviation against log class frequency under linear scalarization. A clearly negative slope confirms the premise; a flat one refutes it. The second is the same regression under the constrained method, which should flatten the slope, together with the per-class versus shared-multiplier ablation, which tests whether rarity-awareness is the cause. The third is the matched-compute comparison against filtering and against tuned linear weights on gold-set tail AP, which determines whether constrained fine-tuning is actually necessary. Learned multipliers should also be larger for rarer classes. If they are not, the mechanism we propose is not what is driving any gain we observe.
References
- GARDO: Reinforcing Diffusion Models without Reward Hacking
- Advancing Text-Driven Chest X-Ray Generation with Policy-Based Reinforcement Learning
- Optimizing 3D Diffusion Models for Medical Imaging via Multi-Scale Reward Learning
- Doctor Approved: Generating Medically Accurate Skin Disease Images through AI-Expert Feedback
- Safe RLHF: Safe Reinforcement Learning from Human Feedback
- Scalar reward is not enough: a response to Silver, Singh, Precup and Sutton (2021)
- Checklists Are Better Than Reward Models For Aligning Language Models
- Causal Disentanglement for Robust Long-tail Medical Image Generation
- Long-Tailed Classification of Thorax Diseases on Chest X-Ray: A New Benchmark Study
- Rethinking Perceptual Metrics for Medical Image Translation
- Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challenge