Abstract
In this work, we present an end-to-end framework for camera-based 3D multi-object tracking, called DQTrack. To avoid heuristic design in detection-based trackers, recent query-based approaches deal with identity-agnostic detection and identity-aware tracking in a single embedding. However, it brings inferior performance because of the inherent representation conflict. To address this issue, we decouple the single embedding into separated queries, i.e., object query and track query. Unlike previous detection-based and query-based methods, the decoupled-query paradigm utilizes task-specific queries and still maintains the compact pipeline without complex post-processing. Moreover, the learnable association and temporal update are designed to provide differentiable trajectory association and frame-by-frame query update, respectively. The proposed DQ-Track is demonstrated to achieve consistent gains in various benchmarks, outperforming previous tracking-by-detection and learning-based methods on the nuScenes dataset.
Introduction: The Representation Conflict
Multi-object tracking (MOT) in 3D space is a cornerstone of autonomous driving. Traditionally, this was achieved via a "tracking-by-detection" pipeline: a detector identifies objects in each frame, and hand-tuned heuristics—such as Kalman filters and Hungarian matching—link these detections across time. While functional, these pipelines are often brittle and difficult to optimize end-to-end.
Recent "tracking-with-query" methods attempted to streamline this process using Transformers. However, these methods typically employ a single query embedding to handle both detection (identifying an object's presence) and tracking (maintaining its identity). The authors identify a fundamental representation conflict in this approach: detection must be identity-agnostic, focusing on general class features, whereas tracking must be identity-aware to distinguish between similar-looking objects. Forcing both tasks into a single embedding leads to sub-optimal performance.
The DQTrack Framework
To resolve this conflict, DQTrack introduces the Decoupled-Query paradigm. Instead of a unified query, the system utilizes two distinct types:
- Object Queries ($Q_O$): Dedicated to detecting new objects in the current frame.
- Track Queries ($Q_T$): Responsible for maintaining the identity and location of objects already being tracked.
This separation allows each query type to specialize, avoiding the performance degradation caused by conflicting objectives.
Key Components
To integrate these queries into a differentiable, end-to-end pipeline, DQTrack employs two novel modules:
- Learnable Association: Rather than relying on fixed metrics like 3D IoU or Euclidean distance, this module uses a neural network to compute an affinity matrix. By combining appearance features from the Transformer with motion features (BEV positions), the model learns the optimal strategy for linking new detections to existing trajectories.
- Temporal Update: To maintain track query accuracy over time, the framework uses an Exponential Moving Average (EMA) to update track features frame-by-frame. This ensures the "memory" of an object remains robust as its appearance or motion evolves.
Experimental Results
DQTrack was evaluated on the nuScenes dataset, a large-scale benchmark for 3D perception in autonomous driving. The model achieved a new state-of-the-art 52.3% AMOTA (Average Multi-Object Tracking Accuracy) on the test set, significantly outperforming previous methods such as MUTR3D (27.0%) and PF-Track (43.4%).
Ablation studies confirmed the effectiveness of the decoupled approach. The "Learnable Association" module alone provided a 0.7% AMOTA boost over standard matching, demonstrating that the network successfully learns superior identity-linking strategies from data. Despite its dual-query architecture, DQTrack remains efficient, running at approximately 6 FPS with a V2-99 backbone.
Conclusion
By decoupling detection and tracking into specialized queries, DQTrack overcomes the representation conflict inherent in previous Transformer-based trackers. This end-to-end framework eliminates the need for complex post-processing while achieving superior accuracy on competitive benchmarks. Future work may explore extending this paradigm to multi-modal sensor setups and improving long-term re-identification for occluded objects.
Summary updated