CRACT: Cascaded Regression-Align-Classification for Robust Tracking

Heng Fan, Haibin Ling

Abstract

Modern visual tracking often relies on parallel refinement of object proposals, which can lead to feature misalignment and vulnerability to distractors. This paper presents CRACT, a tracker featuring a novel Cascaded Regression-Align-Classification (CRAC) module. By sequentially performing box regression, feature alignment, and classification, CRACT ensures that the classifier operates on accurate, noise-free features. Coupled with a multi-scale Pyramid RoIAlign and a dual-component classification system (offline identification and online discrimination), CRACT achieves state-of-the-art performance across seven major benchmarks, including OTB-2015 and UAV123, while maintaining real-time performance at 28 FPS.

The Misalignment Problem in Visual Tracking

In typical Siamese-based tracking architectures, a Region Proposal Network (RPN) generates initial target candidates. These proposals are then refined through two parallel branches: box regression and box classification.

Comparison between standard separated architecture and the proposed cascaded architecture.

The authors identify two critical flaws in this parallel approach:

  1. Feature Contamination: Since classification occurs simultaneously with regression, the features for classification are extracted from the initial, often inaccurate, proposal locations. This misalignment introduces background noise, degrading classification accuracy.
  2. Distractor Vulnerability: Many trackers rely solely on offline template matching, making them susceptible to "distractors"—background objects that resemble the target.

The CRACT Framework

To address these issues, CRACT introduces the Cascaded Regression-Align-Classification (CRAC) module. Instead of parallel branches, CRAC employs a three-step sequential pipeline.

Detailed view of the CRAC module (left) and a visualization of the feature alignment process (right).

Sequential Refinement (CRAC)

  • Box Regression: The module first predicts offsets to adjust the initial proposal's scale and position.
  • Feature Alignment: This is the critical "bridge." The regression offsets are used to "re-align" the features. By re-sampling features from the adjusted bounding box, the model ensures the classifier receives the most accurate representation of the target.
  • Dual-Component Classification: Classification is split into an Identifier and a Discriminator. The Identifier uses a relation network for stable offline template matching. The Discriminator is a lightweight network updated online to learn the current background and suppress distractors.

Pyramid RoIAlign (PRoIAlign)

Standard RoIAlign pools features to a single fixed size. CRACT proposes Pyramid RoIAlign, which pools features at three scales (6x6, 3x3, and 1x1). Concatenating these scales allows the model to capture both fine-grained local details and broader global context, improving robustness against large deformations.

Experimental Results

CRACT was evaluated on seven diverse benchmarks, including OTB-2015, UAV123, NfS, VOT-2018, TrackingNet, GOT-10k, and LaSOT.

  • State-of-the-Art Performance: On OTB-2015, CRACT achieved a Success (SUC) score of 0.726, a 1.4% improvement over SiamAttn. It also led on UAV123 (0.664 SUC) and GOT-10k (0.620 AO).
  • Ablation Studies: The importance of the cascaded architecture was confirmed by removing the alignment step, which resulted in a performance drop from 0.726 to 0.713 SUC on OTB-2015.
  • Efficiency: Despite its complexity, CRACT runs at 28 FPS on a single GTX 1080 GPU, meeting real-time requirements.

Conclusion

CRACT demonstrates that the sequence of refinement operations is as important as the operations themselves. By bridging regression and classification with a feature alignment step, CRACT significantly improves tracking robustness. Future work may involve integrating pixel-wise mask segmentation into this cascaded framework to further enhance accuracy for complex object shapes.

Summary updated

Abstract

High quality object proposals are crucial in visual tracking algorithms that utilize region proposal network (RPN). Refinement of these proposals, typically by box regression and classification in parallel, has been popularly adopted to boost tracking performance. However, it still meets problems when dealing with complex and dynamic background. Thus motivated, in this paper we introduce an improved proposal refinement module, Cascaded Regression-Align-Classification (CRAC), which yields new state-of-the-art performances on many benchmarks.First, having observed that the offsets from box regression can serve as guidance for proposal feature refinement, we design CRAC as a cascade of box regression, feature alignment and box classification. The key is to bridge box regression and classification via an alignment step, which leads to more accurate features for proposal classification with improved robustness. To address the variation in object appearance, we introduce an identification-discrimination component for box classification, which leverages offline reliable fine-grained template and online rich background information to distinguish the target from background. Moreover, we present pyramid RoIAlign that benefits CRAC by exploiting both the local and global cues of proposals. During inference, tracking proceeds by ranking all refined proposals and selecting the best one. In experiments on seven benchmarks including OTB-2015, UAV123, NfS, VOT-2018, TrackingNet, GOT-10k and LaSOT, our CRACT exhibits very promising results in comparison with state-of-the-art competitors and runs in real-time at 28 fps.
BibTeX
@inproceedings{fan2021cract,
  author = {Heng Fan and Haibin Ling},
  title = {{CRACT: Cascaded Regression-Align-Classification for Robust Tracking}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  doi = {10.1109/iros51168.2021.9636803},
}

Loading related content...

Figure