A chest X-ray flattens the whole thorax onto one plane. A radiologist who sees an opacity on the frontal and side views still has to work out, mentally, which lobe it sits in and whether it lies toward the front or the back of the chest. Computer vision already has tools for this kind of problem. "2D-to-3D feature lifting" means taking features computed on flat images and placing them into a 3D coordinate grid using the camera geometry, and these methods reason about space far better than models that stay in 2D. In medical imaging, however, they have been trained and tested almost entirely on synthetic radiographs rendered from CT scans with perfect, fixed geometry. Real clinics add rotated patients, portable machines at odd distances, and erect patients whose organs sit differently than in a supine CT. Clinical AI shows a consistent pattern: models that look strong on curated historical data lose accuracy on the live, uncurated stream of cases. Lifting methods should be especially exposed to this gap, because their central assumption, known geometry, is the one live practice breaks.
We propose to test this directly, and to test whether it can be fixed. We will build a lifting model that estimates each exam's projection geometry instead of assuming it, and we will run it in silent prospective deployment. Silent deployment means the model runs on every incoming frontal-plus-lateral chest radiograph and logs its output without showing it to anyone. The reference standard comes from something hospitals already produce: many patients receive a chest CT within days of their radiograph, and that CT gives 3D ground truth for where the finding actually is. The core claim is that explicit, per-exam geometry makes a lifted 3D model lose less accuracy from retrospective to prospective data than 2D models that learn view-specific shortcuts. A second claim is that a retrospective "geometry stress test" can forecast the size of that loss before deployment.
Core Claim
The claim is specific. On the task of placing a chest finding into a lobe, a front-to-back position, and an approximate 3D centroid from PA and lateral radiographs, the prospective accuracy drop for a geometry-estimating lifting model will be measurably smaller than for (a) a fixed-geometry lifting model and (b) a strong 2D multi-view model trained on the same CT-derived labels. The drop will also be predictable from how each model degrades under simulated rotation and source-distance changes, once those changes are weighted by the geometry distribution actually measured in the prospective stream. If the 2D model degrades just as gracefully, the case for lifting in this setting fails, and we will report that.
This differs from the closest existing work in both directions. X2CT-GAN reconstructs whole CT volumes from two orthogonal views of synthetic radiographs. It reports PSNR of 26.19 dB and SSIM of 0.656 on LIDC-IDRI, and it assumes an exact 90° geometry. It never tests whether findings survive reconstruction, and it never sees a real patient. DecGAN and the PAXRay++ pipeline of Accurate Fine-Grained Segmentation of Human Anatomy in Radiographs via Volumetric Pseudo-Labeling use CT to supervise 2D models. The first decomposes radiographs into tissue layers. The second projects 3D anatomy labels down to 2D masks, reaching 0.93 mIoU frontal agreement. Both stay in 2D, both are retrospective, and the latter explicitly leaves the supine–erect posture gap unmodeled. On the clinical side, the prospective and reader studies in the file, Prospective implementation of AI-assisted screen reading and AI support for accurate and fast radiological diagnosis of COVID-19, operate on native 2D mammograms or native 3D CT, never on lifted representations. We ask a narrower question than "is 3D reconstruction useful": we ask whether an explicit geometric model is what keeps spatial reasoning intact when the geometry stops being clean.
Research Program
Model design. The model follows the 3D-to-2D querying idea of Zhang et al.. Rather than pushing a dense feature frustum into 3D, a fixed grid of query points in a canonical thoracic coordinate frame samples features from the two radiographs. The work on that line reports this approach to be more robust to depth noise and lighter on memory. Each query point $x$ is mapped to image coordinates by
$$u_v \sim K_v \,[R_v \mid t_v]\, x,$$
where $v$ indexes the view (PA or lateral), $K_v$ is the per-exam projection matrix encoding source-to-detector distance and principal point, and $R_v, t_v$ encode patient rotation and offset relative to the canonical frame. The one addition that matters is a small geometry head. This network regresses $K_v, R_v, t_v$ for each exam. It is supervised on synthetic radiographs rendered with randomized geometry, and at test time it runs on the real images. Features sampled at $u_v$ from a shared 2D encoder are fused per query point. A 3D head then predicts, per anatomical region, whether a finding is present and where its centroid lies.
Training data. We render digitally reconstructed radiographs, meaning simulated X-rays computed by summing CT intensities along rays, from LIDC-IDRI and other thoracic CT collections. Rendering randomizes rotation, source distance, and a crude diaphragm and organ shift as a partial stand-in for posture. Lobe and anatomy labels come from a 3D segmentation ensemble, following the PAXRay++ recipe. We then fine-tune on a retrospective set of real radiograph–CT pairs from our institution, taken within a short interval, with finding locations taken from the CT.
Geometry stress test. For every model, we sweep rotation angle, source distance, and view non-orthogonality on held-out synthetic radiographs and on the real retrospective pairs. This produces a curve of accuracy versus each geometric factor. We then write the forecast prospective accuracy as
$$\hat{A}_{\text{pro}} = \int A(g)\, p_{\text{pro}}(g)\, dg,$$
where $A(g)$ is retrospective accuracy at geometry $g$ and $p_{\text{pro}}(g)$ is the distribution of geometry measured on the prospective stream. That distribution is estimated by the geometry head and spot-checked by manual annotation of rotation from clavicle and spine landmarks. The forecast is pre-registered before the prospective data are unblinded.
Prospective silent deployment. All models run in shadow on consecutive PA-plus-lateral studies for a fixed accrual window, with no exclusion for image quality. Portable single-view films are logged separately as an out-of-scope stream. The reference standard is a CT acquired within a pre-specified window, read by an automated pipeline and adjudicated by a radiologist blinded to model output. Because CT is ordered selectively, we will report the referral pattern and analyze verification bias explicitly rather than hide it.
Baselines
The naive reference point is a model that ignores the images beyond presence of a finding. It predicts the most frequent lobe and front-to-back position for each finding type, using the training prevalence. Any method that fails to beat it on the prospective stream has learned nothing spatial.
The strongest 2D competitor is a multi-view 2D model. It is a shared encoder applied to PA and lateral views, with features pooled and a classification head predicting lobe, front-to-back position, and centroid regression. This mirrors the view-averaging design of Wu et al., which reached a standalone AUC of 0.895 in screening mammography without any 3D representation. It is the most serious threat because it receives exactly the same CT-derived 3D labels. The only thing it lacks is explicit geometry. It will use the same encoder, the same rendered and real training data, the same geometric augmentation, and the same hyperparameter search budget as our model. A second 2D baseline adds PAXRay++-style projected anatomy masks as auxiliary supervision, representing the "distill 3D into 2D" school.
The strongest existing lifting method is a reconstruction-first pipeline in the style of X2CT-GAN. It reconstructs a CT volume from the two views and runs a 3D localizer on the result. We retrain it on our data with its published loss design. The only reported numbers on Lacuna for this family are the fidelity figures above, 26.19 dB PSNR and 0.656 SSIM on LIDC-IDRI. There is no reported result found on Lacuna for lobe-level or centroid localization of findings from radiographs by any method, so our baselines must establish the first such numbers on shared data.
The key ablation removes the geometry head and fixes $K_v, R_v, t_v$ to the idealized orthogonal setup. Everything else stays identical, including training on randomized-geometry renders. A second ablation swaps querying for ray-smear lifting in the style of Cao et al., to check whether the lifting mechanism matters beyond geometry estimation.
We will count our model as beating a baseline only if two conditions hold. First, its prospective accuracy must exceed the baseline's by more than the bootstrap confidence interval, on the pre-specified primary metric (lobe-level macro accuracy). Second, its retrospective-to-prospective drop must be smaller than the baseline's.
Why not just…?
Why not just use a 2D multi-view network trained on CT-derived labels? This is the real threat. We do not know it fails. On clean retrospective pairs it may well match the lifting model, since the labels carry all the 3D information. Our bet is narrower: a 2D model learns correlations tied to the typical view geometry, so it should lose more accuracy as rotation and source distance drift. The 3D segmentation results in 3DMV, where backprojection lifted ScanNet accuracy from 52.8% to 75.0%, suggest explicit geometry helps spatial tasks. That evidence comes from calibrated indoor scenes, though, not radiographs. The geometry stress sweep and the prospective drop comparison settle the question directly. If the 2D curve is as flat as ours, we will conclude lifting is unnecessary for this task.
Why not just distill CT knowledge into 2D, as DecGAN and PAXRay++ do? These methods improve 2D classification, with DecGAN gaining AUC on 11 of 14 diseases, and 2D anatomy segmentation. Neither outputs a front-to-back position or a 3D centroid, which is exactly what the synthesis in Volumetric Radiology AI in the Era of Multimodal Large Language Models calls the representational mismatch of 2D-native models. We include the mask-supervised 2D model as a baseline, so this alternative gets a fair test instead of a dismissal.
Why not reconstruct a CT and let existing 3D tools do the rest? Full reconstruction spends model capacity on plausible texture everywhere. The file's gap analysis notes that high SSIM is compatible with erasing small, low-contrast findings. Reconstruction also inherits a fixed-geometry assumption. The reconstruction-first baseline measures this directly on localization accuracy.
Why not just order a CT when 3D information is needed? For many patients that is the right answer. The use case here is triage: deciding which radiographs warrant CT, and giving a spatial prior when CT is delayed or unavailable. We make no claim that lifting replaces CT. The same CTs that clinicians already order are what make the prospective evaluation possible.
Why not just simulate drift retrospectively instead of deploying? This is partly what we propose. The forecast $\hat{A}_{\text{pro}}$ is exactly a retrospective simulation. What the file argues, through the sepsis trigger of Horng et al. and the curated-versus-real label gap in Kim et al. (external AUC 0.883–0.913 on clean labels against 0.829–0.835 on real-world labels), is that such simulations cannot be trusted until they have been checked against a live stream. The silent deployment is that check. If the forecast lands within its confidence interval, later lifting models can rely more on the stress test.
Proposed Experiments and Decisive Checks
The first check is retrospective. We plot accuracy against rotation, source distance, and view non-orthogonality for every model on synthetic and real pairs. The prediction is that the geometry-estimating model has the flattest curves, the fixed-geometry ablation degrades sharply beyond modest rotation, and the 2D model falls in between. If the 2D model is as flat as ours, the core claim is already in doubt before deployment.
The second check is prospective. On the silent stream we report lobe-level macro accuracy (primary), front-to-back accuracy, centroid error in millimetres against the CT, and calibration. Each metric is broken down by measured rotation and by patient subgroups such as lines and tubes, prior surgery, and poor inspiration, which retrospective cohorts tend to underrepresent. The decisive quantity is each model's retrospective-minus-prospective drop, compared across models with paired bootstrap intervals.
The third check tests the forecast. We compare pre-registered $\hat{A}_{\text{pro}}$ with observed prospective accuracy for each model. If the forecast misses badly for all models, the gap is driven by something other than geometry, perhaps posture, case mix, or label noise. We would then report that as a negative finding about where lifting models break, and redirect the method toward posture modeling rather than geometry estimation.
A small follow-on reader study is justified only if the prospective results favor lifting. It would show radiologists the predicted lobe and 3D location during a subsequent live period and measure reading time and localization agreement, following the design logic of Meng et al.. This addresses the separate question of whether a spatial suggestion helps or distracts. We would not make any claim about clinical utility before that study is done, consistent with the claim–design–validation mapping in the volumetric radiology synthesis.
References
- X2CT-GAN: Reconstructing CT From Biplanar X-Rays With Generative Adversarial Networks
- Encoding CT Anatomy Knowledge for Unpaired Chest X-ray Image Decomposition
- Accurate Fine-Grained Segmentation of Human Anatomy in Radiographs via Volumetric Pseudo-Labeling
- Prospective implementation of AI-assisted screen reading to improve early detection of breast cancer
- AI support for accurate and fast radiological diagnosis of COVID-19: an international multicenter, multivendor CT study
- Deep Neural Networks Improve Radiologists’ Performance in Breast Cancer Screening
- MonoScene: Monocular 3D Semantic Scene Completion
- 3DMV: Joint 3D-Multi-view Prediction for 3D Semantic Scene Segmentation
- Volumetric Radiology AI in the Era of Multimodal Large Language Models
- Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning
- Deep learning-based classification of diffusion-weighted imaging-fluid-attenuated inversion recovery mismatch