For decades, the standard pipeline for medical image analysis followed a rigid, three-step sequence: first, an expert or algorithm would segment the organ of interest (drawing a boundary); second, they would calculate geometric properties (like area or diameter); and third, they would apply a formula to derive a clinical metric (like heart volume or fetal weight).
While intuitive, this "segmentation-first" approach has a major flaw: it is a fragile chain. If the segmentation step fails due to image noise or a fuzzy boundary, the error propagates and often amplifies through the subsequent calculations. Researchers are now shifting toward a "direct" paradigm—mapping raw image pixels directly to clinical measurements using end-to-end regression.
The Problem of Cumulative Error
In traditional pipelines, the segmentation mask acts as a bottleneck. As Xiantong Zhen and Shuo Li argue in Towards Direct Medical Image Analysis without Segmentation, these intermediate geometric steps are often "error-prone." For instance, when measuring the right ventricle of the heart—a structure with complex, irregular geometry—tracing a precise boundary is notoriously difficult. A small mistake in the boundary trace leads to a large error in the final volume calculation.
Direct methods bypass this by treating the image as a global pattern-matching problem. Instead of asking "Where is the edge of the heart?", the model asks "What does a 150mL heart look like in this modality?" By training on raw pixels and final clinical labels, the model learns to ignore local noise that might confuse a segmentation algorithm.
From Geometry to Global Pattern Matching
The intuition behind direct estimation is that the information required to calculate a metric is often distributed across the entire image, not just at the boundaries.
Early successes in this area used Regression Forests, as seen in the work of Xiantong Zhen for bi-ventricular volume estimation. However, the field has moved toward deep learning architectures that can fuse multiple views. Gongning Luo and Kuanquan Wang demonstrated that Multi-Views Fusion CNNs can achieve correlation coefficients exceeding 0.97 for cardiac volumes, while being up to 50 times faster than segmentation-based methods.
This approach has proven effective across diverse clinical domains:
- Fetal Health: Szymon Plotka developed BabyNet, which uses a Residual Transformer Module to predict birth weight directly from ultrasound videos, avoiding the accumulation of errors from manual biometric measurements.
- Oncology: The ECTIL model, described by Yoni Schirris, regresses tumor-infiltrating lymphocyte percentages directly from pathology slides, matching expert accuracy with 100x fewer manual annotations.
- Pulmonary Function: The BeyondCT model by Kaiwen Geng maps 3D CT scans directly to airflow metrics, identifying structural correlates of lung capacity that are invisible to traditional geometric analysis.
Robustness to Real-World Noise
One of the most compelling reasons for direct estimation is its resilience. In clinical settings, images are rarely perfect; they suffer from motion artifacts, low contrast, and varying hardware calibrations.
Conor M. Artman introduced SGS EchoNet, a model that estimates ejection fraction from ultrasound without segmentation. In A robust and data-efficient deep learning model for cardiac assessment without segmentation, Artman demonstrates that by replacing explicit outlines with a latent feature regression, the model becomes significantly more robust to the "noise" of real-world clinical environments. Because the model isn't "looking" for a specific line to draw, it doesn't "break" when that line is obscured.
Uncovering Hidden Phenotypes
Perhaps the most surprising result of the direct paradigm is its ability to find "hidden" information. Because these models are not constrained by human-defined geometric proxies, they can discover features that humans don't know how to measure.
In Deep learning interpretation of echocardiograms, Amirata Ghorbani and colleagues at Stanford showed that EchoNet could regress systemic phenotypes—like a patient's age, sex, and height—directly from cardiac ultrasound videos. These are variables that even expert cardiologists cannot reliably estimate by eye, proving that direct regression can extract physiological signals that bypass traditional anatomical boundaries entirely.