Abstract
Survival analysis is a cornerstone of risk assessment in fields ranging from medicine to insurance. However, standard models often fail to account for reporting delays—the time elapsed between an event's occurrence and its official recording. This paper introduces a novel framework that jointly models event timing and reporting delays. By treating unreported events as latent variables and employing an Expectation-Maximization (EM) algorithm, the authors develop a robust estimation method. Furthermore, a two-stage transfer learning approach is proposed to handle "administrative censoring" in newly enrolled cohorts. Experimental results on synthetic and real-world medical datasets demonstrate that the proposed method significantly outperforms naive approaches, which tend to drastically underestimate risk.
Introduction
In traditional survival analysis, researchers observe the time until a specific event occurs. A critical assumption is that once an event happens, it is immediately recorded in the dataset. In reality, many industries face significant reporting delays. For instance, in insurance, a car accident might occur today, but the claim might not be filed for several weeks.
When analyzing data at a fixed point in time, an individual with no recorded event might truly be event-free, or they might have experienced an event that has not yet been reported. Ignoring this distinction leads to "optimistic" bias, where the frequency of events is systematically underestimated. This problem is exacerbated for new cohorts—groups that have recently entered the system. Because their observation window is short, a large proportion of their potential events are likely trapped in the reporting delay phase, a phenomenon known as administrative censoring.
Methodology
The core of the proposed framework is a joint survival model that explicitly accounts for the sequential nature of events and reports.
Joint Modeling of Sequential Processes
The model defines two primary time variables: $T_1$, the time from enrollment to the accident, and $T_2$, the time from the accident to the report. The total time to report is $T = T_1 + T_2$. Since the status of an unreported individual is unknown, it is treated as a latent variable.
EM Algorithm and Two-Stage Estimation
The authors utilize an Expectation-Maximization (EM) algorithm to find the maximum likelihood estimates. In the E-step, the model calculates the conditional probability that an accident occurred given the observed censoring time. In the M-step, the model parameters are updated to maximize the expected log-likelihood based on these probabilities.
To address data sparsity in new "target" cohorts, a two-stage estimation strategy is employed. In the first stage, the baseline hazard (the general risk profile over time) and the reporting delay distribution are estimated from a "source" domain—an older, established cohort with mature data. In the second stage, these parameters are transferred to the target domain, where only the specific covariate effects are estimated. This allows for accurate risk assessment even with very limited follow-up time.
Experiments and Results
The model was evaluated using a synthetic toy model and two benchmark medical datasets, Dialysis and Support2, adapted to simulate insurance-like reporting delays.
The synthetic experiments highlighted the danger of ignoring reporting delays. A "Naive" model, which treats unreported events as non-events, underestimated the risk parameter $\gamma$ by up to 87% when delays were substantial. In contrast, the proposed method ("Ours") consistently recovered parameters close to the ground truth, as shown in the table above.
On the medical datasets, the authors simulated an insurance scenario where they calculated the difference between predicted benefits and collected premiums. The proposed method maintained a near-zero balance, indicating accurate pricing. The Naive method showed a significant negative bias, which in a real-world insurance context would lead to severe financial losses due to underpriced risk.
Conclusion
This research provides a mathematically rigorous and practically applicable framework for survival analysis in the presence of reporting delays. By acknowledging the "hidden" nature of unreported events and leveraging transfer learning, the model enables organizations to perform accurate risk assessments much earlier than previously possible. Future research will focus on relaxing the independence assumption between event time and reporting delay, as well as exploring non-parametric modeling of the delay distribution to enhance flexibility.
Summary updated