TL;DR
Standard metrics like AUC and the C-index are "blind" to the timing of cancer, allowing models to achieve high scores while being no better than a coin flip at predicting when a diagnosis will happen. This paper exposes this "timing collapse" in state-of-the-art models and proposes a new two-stage architecture that successfully separates the risk of developing cancer from the predicted timeline of its onset.
What Problem Does This Paper Address?
In breast cancer screening, the goal isn't just to identify high-risk women, but to determine when they should be screened. A model that predicts a woman will develop cancer in five years should lead to a different clinical path than one predicting cancer in six months.
Currently, longitudinal deep learning models are evaluated using the C-index (concordance index) and AUC (Area Under the ROC Curve). The authors demonstrate a critical flaw: these metrics are dominated by the model's ability to distinguish "cancers" from "healthy controls." Because case-control pairs vastly outnumber case-case pairs (women who both get cancer but at different times), a model can ignore timing entirely—producing "flat" risk curves—and still appear highly accurate.
The researchers set out to answer: Do current models actually know which cancers come first, or are they just good at identifying who is generally at risk?
Methods
The authors provide both a mathematical proof and a new architectural approach to solve the timing problem:
- Metric Analysis: They introduce $c_{time}$, a version of the concordance index calculated only on pairs of cancer cases. This forces the metric to ignore the easy "case vs. control" distinction and focus strictly on whether the model correctly orders early vs. late diagnoses.
- The "ReLU Collapse" Theory: They analyze the "additive-hazard" heads used in popular models like Mirai. They show that under severe class imbalance (where 97.5% of samples are healthy), the gradients push the model toward zero increments for later years. Because these increments are passed through a ReLU, they can "die," resulting in flat risk curves where every woman is assigned the same temporal risk shape.
- Two-Stage Model: To fix this, the authors propose a breast-level model that decouples the two questions:
- Stage 1 (Whether): Predicts the overall 5-year risk ($\pi$) using a fixed, average risk curve shape.
- Stage 2 (When): Learns a specific "timing score" ($s$) using both current and prior mammograms. This score shifts the risk curve earlier or later without changing the total 5-year risk.
Datasets
The study uses the CSAW-CC dataset (Karolinska Institute, 2008–2015), containing mammograms from over 10,000 women.
- Training/Val/Test: Split 50/20/30 by patient.
- Refinement: The authors specifically excluded "same-year" diagnoses (cancers found during the screening exam itself) to ensure they were measuring prediction of future cancer rather than just detection of existing tumors.
Key Insights and Ideas
- High AUC is Deceptive: Models like LMV-Net achieved a high C-index of 0.75 but a $c_{time}$ of only 0.51. This means that while the model is great at finding at-risk women, it is essentially guessing when their cancer will appear.
- The "Flat Prediction" Trap: If a model gives every woman the same risk curve shape (just scaled up or down), it will have a $c_{time}$ of exactly 0.5. The authors proved that standard AUC and C-index are completely invariant to this "flatness."
- Architectural Separation Matters: By forcing the model to learn "timing" as a separate scalar score in Stage 2, the authors prevented the "ReLU collapse" seen in joint models. This allowed the model to actually utilize longitudinal data (prior exams) to adjust the timeline.
Key Results
The proposed two-stage model outperformed existing state-of-the-art (SOTA) methods in timing while maintaining competitive discrimination:
| Method | C-index (Discrimination) | $c_{time}$ (Timing) |
|---|---|---|
| Random Chance | 0.50 | 0.50 |
| OA-BreaCR | 0.691 | 0.547 |
| LMV-Net | 0.746 | 0.509 |
| Ours (Two-Stage) | 0.745 | 0.581 |
| Mirai (External Weights) | 0.745 | 0.571 |
The authors' model was the only one trained on the CSAW-CC dataset to achieve a $c_{time}$ significantly better than chance (95% CI: 0.52–0.64). It matched the performance of the famous Mirai model's published weights, despite Mirai being trained on a much larger, private dataset.
Limitations and Future Work
- Granularity: The diagnosis year ($\tau$) has annual granularity; more precise timing (months) might reveal more subtle patterns.
- Label Noise: The "time to cancer" is inferred from radiological records, which may contain inherent noise regarding when a tumor actually started.
- Calibration: The study focused on ranking (concordance) rather than calibration (whether the predicted 2% risk actually means a 2% probability).
- Future Directions: The authors suggest evaluating these models on the EMBED dataset and investigating why even the best models struggle to beat a simple "constant year 3" prediction in terms of absolute year error.
Summary updated