Time-To-Inconsistency: A Survival Analysis of Large Language Model Robustness to Adversarial Attacks

Yubo Li, Yubo Li 0004, Ramayya Krishnan, Rema Padman

Introduction

Large Language Models (LLMs) often exhibit high initial performance but can degrade significantly under persistent adversarial pressure. Traditional benchmarks typically provide "snapshots" of accuracy, failing to capture the temporal dynamics of model failure in multi-turn dialogues. This research introduces a survival analysis framework to model "time-to-inconsistency," treating conversational failure as a time-to-event problem. By moving from static accuracy to dynamic risk modeling, we can predict when a model is likely to lose consistency before it actually happens.

Methodology

The study applies three primary survival analysis paradigms to conversational data:

  1. Cox Proportional Hazards (PH): A semi-parametric model estimating how specific features influence the "hazard" or instantaneous risk of failure.
  2. Accelerated Failure Time (AFT): A parametric approach that models how features "stretch" or "shrink" the expected time until failure.
  3. Random Survival Forests (RSF): A non-parametric machine learning ensemble capable of capturing complex, non-linear interactions.

A key innovation is the use of Semantic Drift Features:

  • Prompt-to-Prompt Drift: Measures topic shifts between consecutive turns.
  • Context-to-Prompt Drift: Measures deviation of new input from the entire conversation history.
  • Cumulative Drift: Tracks the total semantic distance traveled from the start of the interaction.

Experimental Setup

The framework was evaluated using the MT-Consistency benchmark, comprising 36,951 turns across 9 state-of-the-art models (including GPT-4o, Claude 3.5 Sonnet, and Llama 3.3 70B). The dataset includes 700 questions across 39 academic subjects. Each interaction begins with a base question followed by up to 8 adversarial follow-ups designed to probe robustness through persistent questioning and emotional appeals.

Analysis of Risk Factors

The analysis reveals distinct patterns in how semantic drift impacts model stability. High Prompt-to-Prompt drift acts as a "shock," significantly increasing the immediate risk of failure. Conversely, Cumulative Drift often exhibits a "protective" effect; models that survive early semantic shifts tend to be more robust in later turns, suggesting a survival bias where the model "adapts" to the adversarial context.

Comparison of Cox and AFT results across different models and semantic drift features.

As shown in the figure above, the impact of these features varies across models, but the general trend highlights the critical role of semantic stability in maintaining conversational consistency.

Performance Evaluation

The AFT models, particularly those using the Weibull distribution, outperformed other paradigms in both accuracy and calibration. The best-performing AFT model achieved a C-index of 0.874, indicating high discriminative power in ranking conversation failure times.

Performance of survival models in predicting LLM inconsistency.

Beyond ranking, these models serve as effective early warning systems. An AFT-based monitor successfully flagged 76% of failing conversations with a median lead time of 2 turns before the actual inconsistency occurred. This predictive capability resulted in a 48% reduction in prediction error (Integrated Brier Score) compared to baseline models.

Conclusion and Future Work

Applying survival analysis to LLM robustness provides a more nuanced understanding of conversational failure than static benchmarks. While this study focused on an 8-turn horizon and binary failure events, future work should explore longer-context interactions and distinguish between different failure modes (e.g., hallucinations vs. safety violations). Integrating these risk monitors into live systems could enable proactive interventions, such as handing off high-risk interactions to human operators before a failure occurs.

Summary updated

BibTeX
@inproceedings{li2026time,
  author = {Yubo Li and Yubo Li and Ramayya Krishnan and Rema Padman},
  title = {{Time-To-Inconsistency: A Survival Analysis of Large Language Model Robustness to Adversarial Attacks}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=88C7vSdn0t},
}
Figure