Abrupt transitions in time series with uncertainties

Bedartha Goswami, Niklas Boers, Aljoscha Rheinwalt, Norbert Marwan, Jobst Heitzig, Sebastian F. M. Breitenbach, Jürgen KurthsNature Communications 2017 · DOI

TL;DR

This paper introduces a novel method for detecting abrupt transitions in time series by representing them as sequences of probability density functions (PDFs) instead of point estimates. This "uncertainty-aware" approach allows for a more robust analysis of complex systems, as demonstrated by its successful application to financial markets, climate data, and paleoclimate records.

What Problem Does This Paper Address?

Detecting sudden shifts or "abrupt transitions" in time series data is crucial across many scientific fields, from climate science to finance. However, existing methods often struggle because they either ignore the inherent uncertainties in the data or make simplifying assumptions about these uncertainties (e.g., independence). This can lead to missed transitions or inaccurate dating of events. The core problem is how to reliably identify these critical shifts when the data itself is noisy and its exact value at any given time is uncertain.

Methods

The paper's key innovation is representing a time series not as a sequence of single values ($x_t$), but as a sequence of probability density functions ($p_t(x)$). Each $p_t(x)$ captures the probability of observing a certain value $x$ at time $t$, thus inherently incorporating uncertainty.

To detect transitions within these PDF sequences, the authors adapt recurrence analysis. Traditionally, recurrence analysis builds a "recurrence matrix" indicating when a system state revisits a previous state. Here, they extend this to calculate the probability of recurrence between time points, even when dealing with PDFs. This forms a recurrence network, where nodes are time points and edge weights represent the probability of recurrence.

The crucial step is then analyzing the community structure of these recurrence networks. Communities in a network represent groups of nodes (time points) that are more interconnected among themselves than with the rest of the network. The borders between these communities are interpreted as abrupt transitions in the time series' dynamics. They use a sliding window approach and statistical tests to identify significant community structures, thereby pinpointing transition points. This method explicitly accounts for uncertainties in the data, making it "uncertainty-aware."

Datasets

The authors applied their method to three distinct real-world datasets:

  • Financial Stock Indices: Daily data for DAX (Frankfurt), NASDAQ-100 (New York), and BSE SENSEX (Mumbai) from 2004-2016. This dataset was used to demonstrate the detection of transitions related to well-known economic events like the "mortgage crisis" and "Eurozone crisis."
  • Equatorial Central Pacific SST Anomalies: Monthly sea surface temperature (SST) anomalies from the Niño 3.4 region, spanning from 1870 to 2012. This dataset was used to study transitions in the El Niño-Southern Oscillation (ENSO) and their relationship with the Pacific Decadal Oscillation (PDO).
  • Paleoclimate Proxy Records: 8¹⁸O proxy records from the Dongge, Tianmen, and Qunf caves in Asia, covering periods of the Holocene. This dataset was used to investigate the synchronicity of weakened Asian summer monsoon (ASM) events with North Atlantic ice-rafting events (Bond events), addressing significant chronological uncertainties inherent in paleoclimate data.

Key Insights and Ideas

  • Uncertainty is Not an Add-on, It's Fundamental: The paper strongly argues that treating uncertainties as secondary or assuming them to be simple and independent is a major limitation of traditional time series analysis. Their PDF-based approach integrates uncertainty from the ground up.
  • PDFs Reveal Hidden Dynamics: Representing time series as PDFs, especially for phenomena like ENSO, shows that the underlying dynamics are often non-Gaussian and change shape over time, which point-based estimates can obscure.
  • Recurrence Networks as Dynamic Maps: The use of recurrence networks, built from PDF-based recurrence probabilities, provides a powerful way to visualize and analyze the complex interdependencies within a time series, especially when uncertainties are present.
  • Community Structure as a Transition Indicator: Identifying distinct communities within these networks offers a robust signal for abrupt shifts in system behavior, overcoming the limitations of simpler change-point detection methods.
  • Validating Long-Standing Hypotheses: The method successfully validates hypotheses in paleoclimatology (ASM-Bond event synchronicity) and climatology (ENSO-PDO phase locking) that were difficult to confirm with previous techniques due to data uncertainties.

Key Results

  • Financial Markets: The approach successfully identified abrupt transitions in stock indices corresponding to major politico-economic events like the "mortgage crisis," "Eurozone crisis," and "Brexit/Grexit" periods.
  • Climate Dynamics: Transitions detected in the Niño 3.4 index were found to be significantly coincident with periods of "phase locking" between ENSO and the PDO, suggesting a deeper interplay between these climate phenomena.
  • Paleoclimatology: For the first time, the study provides an "uncertainty-aware" validation of the hypothesis that weakened Asian summer monsoon events during the Holocene were synchronous with North Atlantic ice-rafting events. This was particularly crucial for datasets with significant dating uncertainties.
  • Robustness: The paper demonstrates that their PDF-based approach provides more reliable and robust transition detection compared to using only the mean time series, especially when dealing with significant uncertainties.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, analyzing complex networks and performing statistical tests on sliding windows can be computationally intensive, especially for very long time series or high-dimensional data.
  • Choice of ε: The parameter ε (defining the "e-neighborhood" for recurrence) is crucial and its selection can influence results. The paper mentions using a bisection routine to achieve a pre-specified link density, but the sensitivity to ε might be a point for further investigation.
  • Future Work: The authors express hope that their framework will "motivate a re-thinking of time series approaches in terms of probabilities rather than point-like estimates." This suggests a broad potential for applying this methodology to a wider range of scientific problems where uncertainty is a key factor. Further exploration into optimizing the PDF estimation and community detection algorithms for different types of data could also be valuable.

Summary updated

Abstract

Identifying abrupt transitions is a key question in various disciplines. Existing transition detection methods, however, do not rigorously account for time series uncertainties, often neglecting them altogether or assuming them to be independent and qualitatively similar. Here, we introduce a novel approach suited to handle uncertainties by representing the time series as a time-ordered sequence of probability density functions. We show how to detect abrupt transitions in such a sequence using the community structure of networks representing probabilities of recurrence. Using our approach, we detect transitions in global stock indices related to well-known periods of politico-economic volatility. We further uncover transitions in the El Niño-Southern Oscillation which coincide with periods of phase locking with the Pacific Decadal Oscillation. Finally, we provide for the first time an 'uncertainty-aware' framework which validates the hypothesis that ice-rafting events in the North Atlantic during the Holocene were synchronous with a weakened Asian summer monsoon.
BibTeX
@inproceedings{goswami2017abrupt,
  author = {Bedartha Goswami and Niklas Boers and Aljoscha Rheinwalt and Norbert Marwan and Jobst Heitzig and Sebastian F. M. Breitenbach and Jürgen Kurths},
  title = {{Abrupt transitions in time series with uncertainties}},
  booktitle = {Nature Communications},
  year = {2017},
  doi = {10.1038/s41467-017-02456-6},
}

Loading related content...

Figure