Computer vision models have become remarkably adept at tracking objects across video frames, provided those objects remain visually recognizable. Traditional Video Object Segmentation (VOS) and tracking architectures operate on an appearance-first paradigm: they extract visual patches or pixel-level embeddings from a reference frame and use localized feature matching to find the object in subsequent frames.
This approach relies on the appearance consistency assumption—the expectation that an object's texture, color, and morphology will remain stable over time. However, this assumption collapses in the real world when objects undergo radical physical transformations. When an egg is cracked, a cucumber is sliced into discs, a piece of paper is shredded, or ice melts, the local visual features are completely destroyed, yet the semantic identity of the object persists.
A growing body of research focuses on breaking this dependency on static appearance, shifting the field toward state-aware, geometric, and spatio-temporal reasoning.
The Failure of Visual Feature Matching
To understand why traditional tracking fails during state changes, consider how deep networks represent visual similarity. Features extracted by convolutional networks or vision transformers are highly sensitive to local textures and boundaries. As a result, they are inherently volatile when subjected to environmental or structural shifts.
Even in simpler scenarios without destructive transformations, tracking systems that rely solely on visual appearance struggle with lighting variations, perspective shifts, and temporary occlusions. As noted by Inkyu Shin et al. in Video-kMaX and Yang Fu et al. in CompFeat, relying strictly on appearance similarity causes tracking mechanisms to lose targets or suffer from frequent identity switches when the visual signal becomes noisy or temporarily absent.
[Traditional VOS]
Reference Frame ---> Extract Patches ---> Match Pixels ---> Success (If appearance is rigid)
---> Failure (If object breaks/melts)
When an object undergoes a non-rigid decomposition or phase transition, the failure is absolute. Pavel Tokmakov et al. demonstrated this limitation in Breaking the "Object" in Video Object Segmentation, showing that state-of-the-art patch-matching models fail completely when physical transformations alter the target's visual signature.
Similarly, the EPIC-KITCHENS VISOR benchmark introduced by Ahmad Darkhalil et al. highlights how standard temporal consistency mechanisms fail during everyday tasks like peeling, dicing, or cooking ingredients.
The problem extends beyond physical destruction to contextual discontinuities; for instance, Hengrui Hu et al. showed in Segment Anything Across Shots that visual feature matching fails across discontinuous film edits and camera cuts where an actor's clothing or grooming changes instantly, exposing a fundamental inability to maintain identity without continuous visual tracking.
Decoupling Tracking from Pixels via Motion and Geometry
To build tracking systems capable of surviving state changes, researchers have sought to decouple object identity from raw pixel appearance. One approach is to elevate motion and spatial trajectories to first-class citizens in the tracking architecture.
Early iterations of this idea combined distinct networks for appearance and motion. For example, Xiantong Zhen et al. in Deep Collaborative Tracking Networks and Jun Xiang et al. in Multiple Object Tracking by Learning Feature Representation and Distance Metric Jointly paired appearance-based convolutional layers with motion networks (such as LSTMs) to predict trajectories. This allowed the tracker to maintain a target's path through cluttered backgrounds or severe body deformations where appearance features became unreliable.
A more radical step is to ignore pixel appearance entirely during the tracking phase. Qihao Liu et al. introduced InstMove, a module that predicts instance-level trajectories using only binary segmentation masks. By training the network to model how object shapes move and deform without looking at their internal textures or colors, the system can track fast-moving and heavily occluded objects even when motion blur or environmental shifts distort their visual embeddings.
[InstMove Architecture]
Video Frames ---> Binary Masks Only ---> Trajectory Predictor ---> Stable Track (Texture Invariant)
Alternatively, the tracking space can be mapped into the 3D physical world. Chenming Zhu et al. developed G$^2$TAM, which uses 3D geometry as an implicit memory buffer. By projecting 2D visual features into a spatially aligned 3D latent space, the model anchors the object's identity to its physical coordinates and geometric structure. This makes the tracker highly resilient to extreme camera perspective shifts and long-term occlusions that would otherwise cause a 2D appearance-based memory system to fail.
Reverse Refinement for Chaotic Transitions
When an object undergoes a highly disordered phase transition—such as glass shattering, ice melting, or smoke diffusing—tracking forward in time becomes mathematically chaotic. The initial state is highly ordered (a single solid object), while the subsequent states are highly disordered (hundreds of scattered fragments).
To address this, Zixuan Chen et al. proposed a "reverse refinement" strategy in M^3-VOS. Instead of trying to track the explosion of fragments forward from the stable initial state, their model, ReVOS, propagates information backward from the stable end-state.
Forward Tracking (Chaotic): [Solid Object] --------> ??? --------> [Scattered Fragments]
Reverse Tracking (Stable): [Solid Object] <-------- Refinement <-------- [Scattered Fragments]
Because the final state contains the complete, accumulated history of the scattered components, working backward allows the model to resolve the identity of the fragments and map them precisely back to the source entity, bypassing the tracking failures that occur when modeling the forward transformation.
High-Level Concepts and Action Prompts
A parallel strategy relies on semantic abstraction: if a model understands what an object is conceptually, or what action is being performed on it, it does not need the pixels to remain consistent.
Hongming Zhang et al. addressed this in Video State-Changing Object Segmentation by using cycle-consistency and representation learning to help networks maintain identity across non-rigid transformations like slicing or pouring. Building on this concept of semantic persistence, Zhixiong Zhang et al. introduced Segment Concept (SeC). This framework leverages Large Vision-Language Models (LVLMs) to construct high-level "object concepts" during video playback. Instead of matching raw pixel values frame-by-frame, SeC tracks the evolving concept of the object, providing a robust prior that stabilizes tracking across severe lighting and perspective shifts.
Taking semantic grounding a step further, Liangyang Ouyang et al. developed ActionVOS, which replaces static object descriptions with human action narrations (e.g., "slice the lemon") as segmentation prompts.
By conditioning the vision model on the action causing the transformation, the network learns to anticipate the structural breakdown of the target. The action prompt serves as a temporal bridge, allowing the model to understand that the incoming stream of disparate visual fragments (the slices) still belongs to the single original entity (the whole lemon).
Key Foundational Work
Deep Collaborative Tracking Networks. (2018) — contributed 1 concept elements to this direction.
CompFeat: Comprehensive Feature Aggregation for Video Instance Segmentation (2021) — contributed 1 concept elements to this direction.
Multiple Object Tracking by Learning Feature Representation and Distance Metric Jointly (2018) — contributed 1 concept elements to this direction.