Observe Anything: Human-Intervened Video Understanding with Adaptive Orbital Memory

Seo-Yeon Choi, Kyungsu LeeOpenReview 2026 · Submission to ICLR 2026 · Rejected · OpenReview

TL;DR

The paper introduces Video Object Observation (VOO), a new task that unifies detection, tracking, and human intervention into a single framework. By proposing VOOV (Video Object Observer with human-interVention), the authors demonstrate how sparse human feedback can be embedded directly into a model's hierarchical memory to achieve robust, real-time, and persistent object tracking even under severe occlusion and drastic appearance changes.

What Problem Does This Paper Address?

Despite the rapid progress in deep learning, fully automated video tracking remains surprisingly fragile. If an object disappears behind a wall for too long (long-term occlusion), changes its appearance significantly (e.g., a person putting on a jacket), or is an "unseen" category the model wasn't trained on, traditional trackers often lose the target and never recover.

While "Human-in-the-Loop" (HITL) systems exist to fix these errors, they are usually "ad-hoc." A human might correct a bounding box in one frame, but the model doesn't "learn" from that correction; it just overwrites the output for that specific moment. The authors argue that for truly reliable video understanding, human interventions must be treated as first-class signals that update the model's internal state, allowing corrections to propagate forward in time.

Methods

The authors propose VOOV, a framework built on three core innovations:

  1. Hierarchical Memory: To balance stability and adaptivity, VOOV maintains three distinct memory modules:
  2. Originate Memory: A static anchor capturing the object's initial appearance.
  3. Sequential Memory: A short-term sliding window that tracks local motion and appearance shifts.
  4. Long-Term Memory: A compressed history of the entire sequence to help re-identify objects after long disappearances.
  5. Orbital Deformable Attention (ODA): Inspired by quantum mechanics, ODA treats an object's location not as a single point, but as a "probability cloud" (a Gaussian distribution). It uses optical flow to evolve this cloud over time, allowing the attention mechanism to search for the object in regions where it is most likely to be, even when the view is noisy or obstructed.
  6. Intervention-Driven Adaptation: This is the "secret sauce." When a human provides a correction (like a new bounding box), VOOV performs a lightweight gradient update on the latent memory embeddings ($\theta_t$) rather than retraining the entire network. This allows the model to immediately "understand" the user's intent and maintain that corrected identity in all subsequent frames.

Datasets

The researchers evaluated VOOV across a wide array of challenging benchmarks to prove its versatility:

  • Video Object Tracking/Detection: YouTube-VOS, ImageNet VID, and MOTChallenge.
  • Autonomous Driving: BDD100K and Cityscapes-VIS.
  • Static Detection (for comparison): PASCAL VOC.
  • Novel Benchmarks: The authors introduced "Observation Bootstrap" and "Open-World Emergence" protocols to test how well the model handles objects it has never seen before during training.

Key Insights and Ideas

  • Feedback as Latent State: The most powerful idea here is that human intervention shouldn't just fix the output; it should update the memory. By doing this, a single click from a user can fix a tracking drift that would otherwise persist for the rest of the video.
  • Probabilistic Motion: Moving away from deterministic "point-based" tracking to a probabilistic "orbital" approach makes the system significantly more resilient to the "jitter" and uncertainty common in real-world video.
  • Class-Agnosticism: Because VOOV relies on its memory modules to define what it is looking at, it can "observe anything." It doesn't need to know that a target is a "car" or a "surfer"; it just needs to know that this visual pattern is what the user wants to follow.

Key Results

  • State-of-the-Art Performance: VOOV outperformed existing detectors and trackers (like YOLOv8, TrackFormer, and SAM2) across all five benchmarks, particularly in "re-detection recall" (the ability to find an object again after it reappears).
  • Real-Time Efficiency: Despite the complex memory architecture, VOOV maintains a throughput of ~25 FPS on a single GPU, making it viable for live applications.
  • Interaction Efficiency (IE): The study found that VOOV scales much better with human help than its competitors. A single correction in VOOV leads to a much larger and more persistent jump in accuracy (mIoU) compared to traditional HITL methods.
  • Failure Recovery: In user studies, VOOV achieved a 90.1% recovery rate for missed re-identifications, compared to just 77-80% for automated baselines.

Limitations and Future Work

While VOOV is a major step forward, the authors note a few areas for improvement:

  • Topology Changes: The current model assumes a persistent, trackable identity. It struggles with "abrupt or topology-changing transformations," such as an object splitting into two pieces, melting, or being consumed.
  • Object Interactions: VOOV currently treats each object's memory independently. It does not yet reason about how different objects might interact or collide within a scene.
  • Human Effort: While VOOV minimizes the need for intervention, it still requires a human to notice and correct failures. Future work could involve using Vision-Language Models (VLMs) to act as "automated supervisors" that trigger these memory updates without human input.

Summary updated

Abstract

Video understanding needs both detecting objects in individual frames and maintaining the object identities across time. However conventional methods separate detection and tracking, leading to failures under long-term occlusion, abrupt appearance changes, and the emergence of novel objects. These challenges are particularly severe in dynamic and open-world environments, where objects frequently disappear, reappear, or evolve in appearance, and once identity is lost, automated systems rarely recover. Consequently, a formulation incorporating human-intervention is required to ensure reliable and adaptive continuity. To alleviate this, we introduce Video Object Observation (VOO), a new task unifying detection, tracking, and hunam-intervention, thereby shifting the focus from frame-level recognition to consistent sequence-level observation. To realize this, in this paper, we propose VOOV (Video Object Observer with human-interVention), the first framework explicitly designed for VOO. VOOV integrates three complementary memory modules, such as Originate, Sequential, and Long Term, that jointly encode semantic identity and temporal context, while an Orbital Deformable Attention mechanism models object motion probabilistically. Sparse human-intervention, including initialization, bounding box correction, and target switching, is systematically incorporated into memory, thereby enabling online adaptation without retraining. Experiments on multiple benchmarks demonstrate that VOOV achieves SotA performance, providing robust and real-time observation across diverse and challenging scenarios.
BibTeX
@misc{choi2026observe,
  author = {Seo-Yeon Choi and Kyungsu Lee},
  title = {{Observe Anything: Human-Intervened Video Understanding with Adaptive Orbital Memory}},
  year = {2026},
  url = {https://openreview.net/forum?id=vYSxXKGD0a},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2026},
}
Figure