Detecting Object Tracking Failure via Sequential Hypothesis Testing

Alejandro Monroy Muñoz, Rajeev Verma, Alexander TimansarXiv 2026 · arXiv

TL;DR

This paper introduces a statistically principled framework for real-time object tracking failure detection by framing it as a sequential hypothesis test. By leveraging e-processes, the proposed method provides mathematically guaranteed control over false alarm rates while ensuring rapid detection of actual tracking failures, all without requiring extra model training or fine-tuning.

What Problem Does This Paper Address?

Real-time visual object tracking is critical for applications like autonomous driving, robotics, and video surveillance. However, trackers frequently drift or fail entirely due to common real-world challenges like sudden occlusions, fast motion, and environmental clutter.

While building better trackers is an active area of research, knowing when a tracker has failed is equally important. Deployed systems currently rely on ad-hoc heuristics—such as thresholding the peak value or entropy of a tracker's internal response map—to trigger alerts. These heuristics lack formal statistical grounding. As a result, systems either suffer from delayed failure alerts or trigger excessive false alarms, which can lead to unnecessary and costly human interventions or system re-calibrations.

Methods

The authors address this gap by reformulating tracking failure detection through the lens of sequential hypothesis testing.

Figure 2. A schematic of the proposed sequential testing framework for object tracking. At any given time step t the current frame Yt is fed to the tracker, whose produced prediction or response map (and potential ground truth information) inform a metric Mt capturing object tracking quality. From it, evidence for or against tracking failure is added to a running measure of evidence (the e-process Xt, Eq. 5) whose growth either remains stable (when tracking is satisfactory) or eventually triggers a failure alert (when exceeding a threshold 1

1. The Hypothesis Pair

The system continuously tests a null hypothesis ($H_0$), which states that the expected tracking quality remains above a user-defined tolerance threshold ($\epsilon$), against the alternative hypothesis ($H_1$), which states that a tracking failure has occurred.

2. E-Processes and Testing-by-Betting

To handle the continuous, real-time nature of video streams, the authors employ e-processes (the sequential counterpart to e-values). Derived from the "testing-by-betting" framework, the e-process ($X_t$) acts like a wealth accumulator.

  • When tracking is good, the system "loses bets," and $X_t$ stays low and stable.
  • When tracking quality drops below $\epsilon$, the system "wins bets," causing $X_t$ to grow multiplicatively.

Using Ville’s inequality, the framework guarantees anytime-valid Type-I error control. This means that regardless of when an engineer decides to stop the video and check the tracker, the probability of a false alarm is strictly bounded below a pre-specified significance level $\alpha$ (e.g., 10%). An alert is officially triggered the moment $X_t \geq 1/\alpha$.

3. Tracking Quality Metrics ($M_t$)

The framework is model-agnostic and can operate in two modes:

  • Supervised: Uses the ground-truth bounding box to calculate a Normalized Generalized Intersection-over-Union (NGIoU).
  • Unsupervised: Relies entirely on the tracker's internal response maps, utilizing metrics like Peak Correlation (PC), Certainty Gain (CG), or Sharpness Gain (SG) to proxy tracking confidence.

4. Learning the Betting Rate ($\lambda_t$)

To ensure the e-process grows as fast as possible during a failure, the betting rate $\lambda_t$ is dynamically updated based on past performance using two algorithms: Approximate GRAPA (aGRAPA) and Scale-Free Online Gradient Descent (SF-OGD).

Datasets

The method was evaluated using two distinct tracking models—Kernelized Correlation Filters (KCF) and Siamese networks (SiamFC)—across four popular video tracking benchmarks:

  • OTB-100
  • LaSOT
  • TrackingNet
  • GOT-10k

Testing was conducted on 50 randomly sampled videos from each dataset to ensure robust evaluation across varying video lengths, frame rates, and tracking challenges.

Key Insights and Ideas

  • Anytime-Valid Inference: Traditional hypothesis testing requires a fixed sample size determined in advance. E-processes allow for "optional stopping," making them uniquely suited for real-time video monitoring where a failure could happen at frame 10 or frame 10,000.
  • The Reactivity vs. Stability Trade-off: The choice of betting optimizer matters. SF-OGD updates adaptively based on the gradient of the latest loss, making it highly reactive to sudden tracking drops. Conversely, aGRAPA relies on running empirical means and variances, making it more conservative with fewer false positives but slightly longer detection delays.
  • The Unsupervised Blindspot: Unsupervised metrics can occasionally fail if a tracker drifts off the target but locks onto a visually similar background object with high confidence. In these cases, the internal confidence metric remains high, delaying or preventing the e-process from triggering an alert.

Key Results

The framework successfully delivered on its mathematical promises across all datasets and tracking models:

Figure 3. An example of effective tracking failure detection for different tracking quality metrics Mt (top) and corresponding e- processes Xt (bottom) leveraging both aGRAPA and SF-OGD bet- ting rates (§ 3.3). The video sample (Car-1, OTB-100) depicts ve- hicle tracking in a traffic sequence, with subsequent tracking drift caused by a motorcycle occlusion. Failure alerts are raised consis- tently across metrics, with supervised (NGIoU) reacting later but more stably, while unsupervised ones tend to more volatility.
  • Strict False Positive Control: With the significance level set to $\alpha = 0.1$, the empirical False Positive Rate (FPR) remained safely below 10% across the board, even hitting 0% on the GOT-10k dataset.
  • Low Detection Delay: Failures were flagged rapidly. For instance, using the SiamFC tracker on OTB-100, the SF-OGD betting strategy detected failures with an average delay of just 25.65 frames.

Limitations and Future Work

The primary limitation of this framework is its dependence on the quality of the underlying tracking metric ($M_t$). If a tracker fails silently—meaning it drifts but its internal response maps remain deceptively sharp—the unsupervised metrics will not register a drop, and the e-process will not grow.

Figure 5. An example of partially unsuccessful tracking failure detection, for different tracking quality metrics Mt (top) and cor- responding e-processes Xt (bottom) leveraging both aGRAPA and SF-OGD betting rates (§ 3.3). The video sample (Sheep-1, La- SOT) depicts animal tracking in a natural environment, with sub- sequent tracking drift caused by rapid object movements. While clearly identified by the supervised signal (NGIoU), unsupervised metrics struggle more with raising a proper alert, peak correlation failing to do so entirely.

Future Directions

  1. Automated Recovery: Integrating this failure detection mechanism directly into tracking pipelines to trigger automatic tracker re-initialization or adaptation routines.
  2. Richer Signals: Exploring learned or multi-modal tracking-quality signals to mitigate the unsupervised blindspots.
  3. Broader Vision Tasks: Extending sequential e-process testing to other streaming computer vision domains, such as multi-object tracking, video anomaly detection, and action recognition.

Summary updated

Abstract

Real-time online object tracking in videos constitutes a core task in computer vision, with wide-ranging applications including video surveillance, motion capture, and robotics. Deployed tracking systems usually lack formal safety assurances to convey when tracking is reliable and when it may fail, at best relying on heuristic measures of model confidence to raise alerts. To obtain such assurances we propose interpreting object tracking as a sequential hypothesis test, wherein evidence for or against tracking failures is gradually accumulated over time. Leveraging recent advancements in the field, our sequential test (formalized as an e-process) quickly identifies when tracking failures set in whilst provably containing false alerts at a desired rate, and thus limiting potentially costly re-calibration or intervention steps. The approach is computationally light-weight, requires no extra training or fine-tuning, and is in principle model-agnostic. We propose both supervised and unsupervised variants by leveraging either ground-truth or solely internal tracking information, and demonstrate its effectiveness for two established tracking models across four video benchmarks. As such, sequential testing can offer a statistically grounded and efficient mechanism to incorporate safety assurances into real-time tracking systems.
BibTeX
@misc{munoz2026detecting,
  author = {Alejandro Monroy Muñoz and Rajeev Verma and Alexander Timans},
  title = {{Detecting Object Tracking Failure via Sequential Hypothesis Testing}},
  year = {2026},
  eprint = {2602.12983},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.12983},
}
Figure