When Sensors Fail: Coverage Auditing with Incomplete Sensor Records

Chung-I Huang, Wen-Yi ChangarXiv 2026 · arXiv

In the world of sensor networks—whether tracking city traffic or air quality—we rely on prediction intervals (PIs) to tell us how much to trust a model's forecast. But what happens when the sensors themselves go dark?

A new paper, "When Sensors Fail: Coverage Auditing with Incomplete Sensor Records," by Chung-I Huang and Wen-Yi Chang, reveals a troubling reality: standard ways of measuring model reliability often hide massive failures that occur exactly when sensors are most unreliable.

TL;DR

The authors introduce a "measurement-record audit" that uses time-uniform confidence sequences to evaluate prediction intervals during sensor outages. Their framework reveals that standard "pooled" metrics often mask severe reliability drops during outages, where ground truth for scoring is also frequently missing.

What Problem Does This Paper Address?

Ideally, if a model provides a 90% prediction interval, the actual value should fall within that range 90% of the time. This is called coverage.

The problem is that researchers usually "pool" all forecasts together to calculate this number. This aggregation can conceal a "coverage deficit" during sensor outages. If a model performs perfectly when sensors are healthy but fails miserably when they are glitchy, the overall average might still look fine.

Worse yet, when a sensor fails, we often don't have the "ground truth" (the actual reading) to check if the forecast was right. This "missing truth" makes it mathematically impossible to verify reliability using standard tools.

Methods

The authors propose a rigorous auditing framework that doesn't just look at averages, but tracks evidence over time:

  • Outage State Categorization: Forecasts are grouped by how long the sensor has been unavailable at the time the forecast is issued (e.g., "short," "medium," or "long" gaps).
  • Confidence Sequences: Instead of standard fixed-sample statistics, they use time-uniform inference. This allows for a valid audit that can be updated as new data arrives without the risk of "cherry-picking" a specific timeframe.
  • Simultaneous Count Bounds: They use a "bounded empirical-Bernstein construction" to track two things at once: how many forecasts were "covered" and how many targets were actually "scorable."
  • Identification Ranges: Since some targets are missing, the authors define a range of possible coverage values rather than a single misleading point estimate.

Datasets

The study evaluates several well-known public datasets used in spatial-temporal forecasting:

  • Traffic Records: METR-LA, PEMS-BAY, and the San Diego subset of LargeST.
  • Air Quality Records: AQI-36 and AQI-437.

Key Insights and Ideas

  • The "Blind Spot" of Missing Data: In the METR-LA dataset, the authors found that for forecasts issued during "long gaps" (outages > 12 steps), only 7.6% actually had an observed target to score against. We are often most blind to model performance exactly when the system is under the most stress.
  • The Danger of Averages: A model can have a marginal coverage of 92% (appearing very reliable) while its coverage during an outage might be as low as 57%.
  • Inconclusive is a Valid Result: In many cases, the audit returned "inconclusive." This isn't a failure of the math; it’s a substantive finding that there simply isn't enough scorable data in these benchmarks to prove the models are reliable during outages.

Key Results

The quantitative results highlight a stark gap between "normal" operation and outage states. As shown in the table below, while marginal coverage often meets or exceeds the 0.90 target, "long-gap" coverage frequently plummets.

Table I: Selected descriptive coverage at nominal 0.90. Note the significant drop in coverage for long gaps compared to marginal averages.

For the AQI-36 dataset, the marginal coverage was a healthy 0.918, but for long-gap outages, it dropped to a dismal 0.575. Across 30 primary audits, 26 were inconclusive, highlighting the severe lack of evidence available for certifying model reliability in real-world, "messy" sensor environments.

Limitations and Future Work

  • Conservative Bounds: Because the audit allows for arbitrary dependencies within "whole-network batches," the resulting bounds can be quite wide (conservative).
  • No Imputation: The method does not attempt to reconstruct or "guess" missing truth; it only audits what is actually there.
  • Root Cause: The audit identifies when coverage fails but doesn't identify the physical cause of the sensor outage (e.g., hardware failure vs. network lag).

The authors suggest that future work should incorporate independent reference measurements or event telemetry to help fill the "evidence gap" identified by this audit.

Summary updated

Abstract

Sensor outages can conceal prediction-interval coverage deficits behind acceptable marginal performance, while unavailable future readings leave some issued forecasts unscorable. We present a measurement-record audit that links issue-time outage duration, original target availability, and time-uniform statistical evidence. The audit retains complete forecast keys, batches all sensors and overlapping horizons at each measurement tick, and projects simultaneous covered-count and scored-count bounds onto a cumulative conditional expected coverage ratio. This target is distinguished from empirical coverage and outage-conditional population coverage. Selected traffic and air-quality results illustrate the gap between pooled and long-outage coverage; in METR-LA, only 7.6% of issued long-gap forecasts have originally observed targets. Of 30 primary sequential outage-state audits, 26 are inconclusive and four lack scored support. This concise presentation gives the core construction and selected aggregate evidence without the full experimental supplement.
BibTeX
@misc{huang2026when,
  author = {Chung-I Huang and Wen-Yi Chang},
  title = {{When Sensors Fail: Coverage Auditing with Incomplete Sensor Records}},
  year = {2026},
  eprint = {2611.11324},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11324},
}
Figure