Monitoring What a Streaming Tracker Is Responsible For

A video tracker running live on a robot or headset is usually built to perform an action: output a mask for this frame. The user's actual goal is a state: keep the named object tracked for the whole stream. When frames are corrupted or the object is hidden for a while, a per-frame tracker keeps emitting masks and keeps propagating its most confident guess. It has no idea whether the thing it was asked to maintain still holds. We propose to give an online tracker an explicit estimate of that fact. The tracker keeps a running probability that "the target is correctly tracked" is still true. The probability is computed by a small filter whose noise parameters are re-estimated online as corruption comes and goes. When the probability drops, the tracker hands the frames to a slower recovery module and does not release the task until the state is verified as restored.

In two sentences: treat "keep the object tracked" as a monitored obligation rather than a stream of independent per-frame outputs, and estimate whether the obligation holds using a filter that learns the current noise level online. This should let the system ignore transient corruption that has not actually broken the track, spend expensive recovery only on real losses, and keep latency near the cheap online model's.

Core Claim

The claim draws on work in agent delegation. That work separates action-based delegation, meaning "do this step," from state-based delegation, meaning "make sure this condition holds." A Model of Delegation for Multi-agent Systems formalizes the split with separate logical operators. Delegation and Responsibility adds "whole-hearted satisfaction": an agent holding a state responsibility must actively monitor the condition and intervene when it is threatened, rather than succeed by luck. Implementing responsibility for states and events builds this into an agent architecture called NoA. In NoA, a violated state condition automatically spawns new repair actions.

All three papers assume the condition can be checked exactly, as a true-or-false predicate. Norman and Reed also note that their semantics is essentially timeless. Our contribution is a probabilistic and timed version of state responsibility for one concrete, measurable setting: online referring video object segmentation, the task of segmenting the object described by a text phrase in each frame as the video arrives. The responsibility holds when the target is inside the predicted mask. The system cannot observe this directly, so it maintains a belief

$$b_t = P\big(z_t \in \mathcal{S} \mid y_{1:t}; \theta_t\big),$$

where $z_t$ is the latent true target state (box and appearance embedding), $\mathcal{S}$ is the set of states in which the current track is correct, $y_{1:t}$ are the tracker's per-frame outputs so far (box, query score, embedding), and $\theta_t$ are observation-model parameters estimated online. The observation model is a two-part mixture:

  • a clean component whose noise covariance is learned;
  • a corrupted component that down-weights the frame.

The tracker escalates to recovery when $b_t < \tau$. It releases the escalation only after $b_t$ has stayed above $\tau$ for a short verification window. Releasing only after verification is the operational form of "the delegate stays accountable for the outcome": recovery is not a fire-and-forget event.

The testable claim is that this belief separates two cases that a raw confidence score conflates. In the first, the frame is noisy but the target is still being tracked, so the right move is to do nothing. In the second, the track is actually lost, so the right move is to repair it. If the claim holds, the system should cut the time spent silently lost under corruption while escalating on fewer frames than a confidence threshold at equal accuracy.

Research Program

The base tracker is OnlineRefer. It segments frame by frame and carries the highest-confidence query from one frame forward as a box, semantic, and positional prior for the next. It outperformed offline clip-based methods on clean benchmarks. Its known weakness is our motivation: greedy propagation of the top query leaves it relying on a stale prior during long occlusion, and nothing in it detects that the prior has gone bad.

Monitoring filter. On top of OnlineRefer we add a monitoring filter over a low-dimensional latent state. The state is the target box plus a projected appearance embedding. Its parameters are fit by online expectation-maximization (EM): alternate between inferring the latent state with the parameters fixed and updating the parameters with the inferred state fixed.

  • Inference step. A fixed-lag Kalman smoother over the last $L$ frames. It is bounded, not global, so latency stays constant.
  • Update step. Recursive refits of the clean-observation noise covariance and the per-frame corruption weight.

The design borrows from two papers:

Recovery module. When escalation fires, a recovery module re-runs segmentation over the last $K$ buffered frames using several top query hypotheses instead of one. It re-scores them against the text embedding and the filter's last trusted appearance state, then reseeds OnlineRefer's propagated query. This is the "sub-delegation" step. Structurally it resembles the split in Dispider, where a light "Watcher" model emits <TODO> tokens that trigger a heavier "Speaker." The difference is where accountability sits. In Dispider the trigger is a learned token probability with no condition being tracked. In our design the trigger comes from an explicit probability of state violation, and the monitor keeps checking after the repair.

Analyses. We will measure three things: how well calibrated $b_t$ is against ground-truth track correctness; how the escalation rate trades off against accuracy as $\tau$ varies; and how sensitive results are to the lag $L$ and the recovery window $K$.

Baselines

Do nothing. Stock OnlineRefer with the Swin-L backbone, run causally on the same corrupted streams. On clean data the file reports 63.5 J&F on Refer-YouTube-VOS and 64.8 J&F on Refer-DAVIS17 for OnlineRefer. J&F averages region overlap and boundary accuracy. On test-time corrupted versions of these benchmarks, no reported result was found on Lacuna for OnlineRefer or any other method, so we will produce those numbers ourselves with identical weights.

Confidence-threshold escalation. This is the strongest competitor and the one most likely to erase our gain. It uses the same recovery module, but triggers when OnlineRefer's top query score falls below a threshold, with the same verification-before-release rule. It gets the same compute and the same budget for tuning thresholds on a held-out validation split. It isolates whether the latent belief adds anything beyond the raw score.

Fixed-noise Kalman monitor. The same filter and escalation rule, but with noise covariance and corruption weight set once on validation data and not updated online. This ablation isolates the online EM ingredient.

Two-step monitor. First estimate a per-frame corruption level from image statistics, then filter with that plugged in. This tests the one-step versus two-step argument from Yang et al. in our setting.

Always-escalate. Run recovery every frame. This is an accuracy-and-cost upper reference.

Offline windowed bidirectional re-segmentation. An oracle that is not latency-feasible, used to show how much of the gap to offline processing we close.

What counts as winning. Beating the confidence-threshold baseline means at least one of two outcomes on corrupted streams: higher J&F at a matched fraction of escalated frames, or equal J&F at a significantly lower escalation fraction. In both cases we also require lower "silent loss time," the frames where the track is wrong and no escalation is active. Significance is assessed over videos and corruption seeds.

Why not just…?

Why not just threshold the tracker's confidence score? This is the real threat. If a well-tuned score threshold matches the belief-driven monitor, then the state-responsibility framing reduces to engineering and we will report that. Our reason to expect a gap is that a single-frame score falls under both transient corruption and true loss, and so triggers repair on noisy-but-fine frames. The belief integrates evidence over time under an estimated noise level. The escalation-rate-versus-J&F curves, stratified by corruption type (transient noise versus sustained occlusion), will settle this directly.

Why not just train with corruption augmentation or an auxiliary noise-prediction head? Enhancing Trajectory Prediction through Self-Supervised Waypoint Distortion Prediction shows that making an encoder predict the added noise yields robust features. It reports up to 82.5% better Final Displacement Error than baselines on distorted TrajNet++ inputs. That approach improves each frame's output but still has no mechanism to notice or repair a lost track. It is also complementary rather than competing. We will fine-tune OnlineRefer with this kind of auxiliary head and run both with and without our monitor. If the robust-trained tracker alone closes the gap, the monitor is unnecessary.

Why not just use a Kalman filter with fixed noise? Corruption in deployment is non-stationary, and a covariance tuned for one regime either over-smooths clean frames or under-reacts to heavy noise. The fixed-noise ablation measures exactly this. If it matches online EM, we drop the EM.

Why not just process short offline windows bidirectionally? Bidirectional clip models are the regime OnlineRefer already outperformed on clean data, and they add buffering latency on every frame. We keep them only as an oracle and as the occasional recovery step.

Proposed Experiments

Data. Refer-YouTube-VOS and Refer-DAVIS17, streamed causally. At test time we apply controlled corruptions: Gaussian and impulse noise, motion blur, frame dropout, and synthetic occluders held over the target for 0.5–3 seconds. We vary severity and a mix of transient versus sustained events. A2D-Sentences and JHMDB-Sentences serve as a secondary check, since the file lists no numbers for them.

Metrics. J&F, silent loss time, fraction of frames escalated, per-frame latency and memory on one GPU, and calibration of $b_t$ as expected calibration error against ground-truth track correctness.

Decisive checks. The first check is the stratified comparison with confidence-threshold escalation. The belief monitor must escalate less on transient corruption and react faster on sustained occlusion, at equal J&F. The second check is that online EM beats fixed noise when severity changes mid-stream. The third check is that on clean streams the monitor rarely fires and J&F stays at the stock OnlineRefer level. That guards against the regression on clean data that noise-robust methods like SSWDP are known to risk when mis-tuned.