Adaptive Change Point Monitoring for High-Dimensional Data

Teng Wu, Runmin Wang, Hao Yan, Xiaofeng ShaoarXiv 2021 · DOI · arXiv

Abstract

In this paper, we propose a class of monitoring statistics for a mean shift in a sequence of high-dimensional observations. Inspired by the recent U-statistic based retrospective tests developed by Wang et al. (2019) and Zhang et al. (2020), we advance the U-statistic based approach to the sequential monitoring problem by developing a new adaptive monitoring procedure that can detect both dense and sparse changes in real-time. Unlike previous methods where self-normalization was used, we instead introduce a class of estimators for the $q$-norm of the covariance matrix and prove their ratio consistency. To facilitate fast computation, we further develop recursive algorithms to improve the computational efficiency of the monitoring procedure. The advantage of the proposed methodology is demonstrated via simulation studies and real data illustrations.

Introduction

In modern industrial applications, such as advanced manufacturing and distributed sensor networks, practitioners frequently encounter high-dimensional data streams where the number of variables ($p$) often exceeds the number of available observations ($n$). A critical task in managing these streams is change point detection: the real-time identification of shifts in the statistical properties of the data, such as a sudden change in the mean.

Existing monitoring frameworks face three significant challenges. First, the Dense vs. Sparse Dilemma: a change may manifest as a small shift across many variables (dense) or a large shift in only a few (sparse). Most existing tools are optimized for one scenario, failing in the other. Second, Cross-Sectional Dependence: variables in high-dimensional settings are rarely independent, and failing to account for these correlations leads to high false-alarm rates. Finally, Computational Overhead: real-time monitoring requires statistics to be updated instantly. Traditional high-dimensional U-statistics involve brute-force calculations that are often too slow for high-frequency production environments.

Methodology

$L_q$-Norm Statistics and Adaptive Combination

To address the detection of varying change types, this research utilizes a family of test statistics based on $L_q$-norms. The $L_2$-norm is particularly effective at detecting dense, widespread changes where the signal is distributed across many sensors. Conversely, higher-order norms, such as the $L_6$-norm, are "sharper" and more sensitive to sparse outliers or shifts affecting only a small subset of the dimensions.

Rather than forcing a choice between these two approaches, the authors propose an adaptive combination ("Comb"). By combining $L_2$ and $L_6$ statistics into a single procedure, the framework maintains high power across a broad spectrum of potential change types. A key theoretical insight supporting this combination is that U-statistics for different $q$ values are asymptotically independent, allowing for a unified test that does not suffer from statistical "double-counting."

Covariance Estimation and Self-Normalization

Previous methodologies often relied on self-normalization, which, while theoretically robust, is computationally intensive. This paper introduces a new class of ratio-consistent estimators for the $q$-norm of the covariance matrix. This approach allows the system to be calibrated using a "Phase I" training period of clean data. By moving away from continuous self-normalization, the authors achieve a significant boost in computational speed and a slight improvement in statistical power.

Recursive Algorithms for Real-Time Deployment

A primary contribution of this work is the development of recursive formulas for updating the monitoring statistics. In high-dimensional settings, brute-force calculations for U-statistics typically scale at $O(n^4 p)$. The proposed recursive algorithm reduces this complexity to $O(n^2 p)$. This optimization is vital for real-time deployment, where the time between observations is minimal.

Experimental Results

Simulation and Speedup

Validation through Gaussian sequences with varying cross-sectional correlations ($ ho$) demonstrated that the combined test consistently performed well, nearly matching the specific power of the $L_2$ test for dense changes and the $L_6$ test for sparse changes. Furthermore, in a test case with $n=100$ and $p=50$, the recursive algorithm proved to be approximately 13 times faster than the brute-force approach, a performance gap that widens as the number of observations increases.

Real-World Applications

The methodology was further tested on two distinct manufacturing datasets:

  1. Tonnage Dataset: Monitoring a multi-channel forging process where four strain gauge sensors measure press force. The goal was to detect process anomalies or missing parts.
  2. Rolling Dataset: Monitoring high-speed steel rolling via downsampled images ($16 \times 64$ pixels, totaling 1,024 dimensions). In this case, the $L_2$ test failed to detect surface defects because the changes were too sparse. However, the $L_6$ and the Combined test successfully signaled an alarm, highlighting the necessity of the adaptive approach in practical settings.

Conclusion

This research provides a robust and computationally efficient framework for monitoring high-dimensional data streams. By leveraging the "blessing of dimensionality"—where larger $p$ can actually improve asymptotic approximations—the proposed adaptive $L_q$-norm procedure effectively bridges the gap between dense and sparse change detection.

Future work will focus on extending this framework beyond mean shifts to detect changes in the covariance matrix (variance/correlation shifts) and developing methods for "localization" to identify which specific variables or sensors are responsible for a detected change.

Summary updated

Abstract

In this paper, we propose a class of monitoring statistics for a mean shift in a sequence of high-dimensional observations. Inspired by the recent U-statistic based retrospective tests developed by Wang et al.(2019) and Zhang et al.(2020), we advance the U-statistic based approach to the sequential monitoring problem by developing a new adaptive monitoring procedure that can detect both dense and sparse changes in real-time. Unlike Wang et al.(2019) and Zhang et al.(2020), where self-normalization was used in their tests, we instead introduce a class of estimators for $q$-norm of the covariance matrix and prove their ratio consistency. To facilitate fast computation, we further develop recursive algorithms to improve the computational efficiency of the monitoring procedure. The advantage of the proposed methodology is demonstrated via simulation studies and real data illustrations.
BibTeX
@misc{wu2021adaptive,
  author = {Teng Wu and Runmin Wang and Hao Yan and Xiaofeng Shao},
  title = {{Adaptive Change Point Monitoring for High-Dimensional Data}},
  year = {2021},
  eprint = {2101.06839},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.06839},
}

Loading related content...

Figure