Introduction
Traditional object detection frameworks, such as YOLO or Faster R-CNN, typically treat detection as a regression and classification problem with fixed-size outputs. This approach necessitates complex, hand-engineered heuristics like anchor boxes and Non-Maximum Suppression (NMS) to handle the variable number of objects in a scene. While effective in 2D, these methods often struggle with video, where temporal consistency is crucial. Most video systems rely on a two-step "detect-then-track" pipeline, which is prone to errors and fails to treat moving objects as continuous entities.
This paper introduces P2S-VID, an extension of the Pix2Seq framework that treats video object detection as a pure language modeling task. By representing video objects as discrete sequences of tokens—referred to as 3D "tubelets"—the authors eliminate the need for traditional heuristics and achieve state-of-the-art results by predicting coordinates across multiple frames in a single unified sequence.
The P2S-VID Framework
The core innovation of P2S-VID lies in its approach to tokenization and temporal modeling. The model "speaks" the language of bounding boxes, converting spatial and temporal data into a discrete vocabulary.
3D Tokenization and the "NA" Token
In the original Pix2Seq model, an object is represented by four coordinate tokens and a class label. P2S-VID extends this to $4N + 1$ tokens, where $N$ represents the number of frames in a video clip. This sequence captures the bounding box coordinates for an object across the entire temporal window.
To handle the dynamic nature of video—where objects may enter, leave, or be occluded—the authors introduced a special "Not Available" (NA) token. This allows the model to maintain a fixed sequence length and represent the absence of an object in specific frames without breaking the temporal structure of the tubelet.
Feature Fusion Architectures
The researchers explored three distinct architectural strategies to integrate temporal information:
- Early Fusion: Utilizing a Video Swin Transformer backbone to extract spatio-temporal features directly from the input volume.
- Middle Fusion: Implementing pairwise compositional cross-attention within the encoder to relate features across frames.
- Late Fusion: Incorporating 3D position embeddings in the decoder to provide the model with a sense of temporal progression during sequence generation.
Overcoming the Classification Gap
A significant challenge identified during development was the imbalance between localization and classification. Because a single object is represented by many coordinate tokens but only one class token, the model's loss function was naturally biased toward localization. To rectify this, the authors implemented Class Weight Equalization, increasing the loss weight of class tokens during training. This adjustment ensured the model remained as accurate in identifying what an object was as it was in determining where it was.
Experimental Results
The model was evaluated across three diverse datasets to test its versatility:
- UA-DETRAC: A complex traffic dataset where P2S-VID achieved a 91.14% mAP (mean Average Precision), surpassing the previous state-of-the-art VSTAM (90.39%) and the static Pix2Seq baseline (88.62%).
- ACAD (Canadian Animal Detection): Focused on wildlife monitoring in natural environments.
- IPSC (Induced Pluripotent Stem Cells): A microscopic dataset for cell tracking, demonstrating the model's efficacy in non-natural visual domains.
In all cases, the video-aware P2S-VID outperformed static baselines, proving that temporal context is vital for resolving visual ambiguities.
Key Insights and Discussion
Objects as Indivisible Tubelets
By conceptualizing video objects as tubelets, the model learns temporal continuity inherently. This removes the brittleness of external tracking algorithms, as the model predicts the entire trajectory of an object as a single coherent output.
The Batch Size and Memory Trade-off
Token-based models are highly sensitive to batch size. The authors found that freezing the backbone (the image-processing component) often yielded better results. This counter-intuitive finding was due to memory constraints; freezing the backbone freed up enough GPU memory to allow for larger batch sizes, which proved more beneficial for the transformer's learning than fine-tuning the visual features.
Redundancy via Sliding Windows
Accuracy was further improved by using a sliding window approach with a small stride. By processing overlapping chunks of video, the model received multiple "looks" at the same object from different temporal positions, leading to more robust final detections.
Limitations and Future Directions
The primary constraint of P2S-VID is computational intensity. As the number of frames ($N$) increases, the output sequence length grows linearly, making the model memory-heavy. Current experiments were limited to short temporal windows (2 to 8 frames).
Future work will focus on scaling the architecture to handle longer sequences, potentially enabling full end-to-end Multi-Object Tracking (MOT) across entire video files. Additionally, the authors are investigating more efficient tokenization methods, such as 1D coordinate tokens, to reduce sequence length without sacrificing the precision of the discrete vocabulary.
Summary updated