TL;DR
This paper tackles the challenging problem of instance segmentation in scenarios with heavy occlusions, particularly in sports like basketball. The authors introduce a novel "location-aware copy-paste" data augmentation technique to generate more realistic training data and a modified Hybrid Task Cascade (HTC) model with a CBSwin backbone and MaskIoU heads to improve performance, ultimately achieving the top rank in the ACM MMSports 2023 challenge.
What Problem Does This Paper Address?
Instance segmentation, the task of identifying and outlining individual objects in an image, is crucial for many computer vision applications. However, real-world scenarios, especially in sports, often involve significant occlusions where objects are partially or fully hidden. Traditional instance segmentation models and evaluation metrics struggle with these occluded instances. The ACM MMSports 2023 challenge specifically highlighted this by introducing a dataset focused on segmenting basketball players and a new "Occlusion Metric" (OM) that prioritizes correctly segmenting these occluded objects. The challenge of a small dataset (324 training images) further exacerbates this problem, making robust data augmentation and specialized model architectures essential.
Methods
The paper proposes a three-pronged approach:
- Location-Aware Copy-Paste Data Augmentation: This is the core novelty. Instead of randomly pasting object instances, this technique first detects the "playable area" of a basketball court using traditional computer vision methods (contour detection, Hough line transform). Then, it strategically pastes extracted human instances (from ground truth masks) within this detected area. To further simulate occlusions, there's a 70% chance an additional entity is pasted on top of an existing one, mimicking realistic overlaps. Photometric and geometric transformations are also applied to the pasted entities.
- Modified Hybrid Task Cascade (HTC) Architecture: The authors adopt the HTC framework, a strong baseline for instance segmentation. They replace the standard ResNet backbone with a CBSwin-Base backbone, aiming for better visual representations and data efficiency. Crucially, they introduce "MaskIoU heads" for each mask head within the HTC. This deviates from standard practice by allowing the model to learn mask quality independently from classification scores. Additionally, they increase the size of the mask features extracted by the RoIPooling layer from 14x14 to 20x20 to improve mask reconstruction, especially in occluded regions.
- Stochastic Weight Averaging (SWA) Training: To enhance model generalization, the authors employ SWA. After the initial training, the model is further trained for an additional 12 epochs, saving a checkpoint at each epoch. The final model's weights are then an average of these 12 checkpoints.
Datasets
- ACM MMSports 2023 DeepSportRadar Dataset: This dataset, featuring basketball court scenes with human subjects, was used for both training and evaluation. It's noted for its modest size (324 training images) and the focus on occlusion scenarios.
- COCO: While not explicitly used for training in this work, the paper references COCO metrics as the traditional standard for object detection and segmentation, highlighting their limitations for occlusion scenarios.
Key Insights and Ideas
- Context Matters for Augmentation: Simply pasting objects isn't enough. By understanding the "playable area" of a basketball court, the copy-paste augmentation becomes much more realistic and effective, leading to better generalization.
- Occlusion Requires Specialized Metrics: The introduction of the Occlusion Metric (OM) is a significant step. It forces researchers to focus on the hardest cases (occluded instances) rather than just overall accuracy, pushing the field forward.
- Mask Quality is Paramount for Occlusions: The MaskIoU head and increased mask feature size demonstrate that improving the model's ability to generate high-quality masks directly addresses the challenge of reconstructing occluded object details.
- SWA for Robustness: Stochastic Weight Averaging is a simple yet effective technique to improve model generalization, especially when dealing with limited data or complex tasks.
Key Results
The paper reports achieving the top-1 position in the ACM MMSports 2023 segmentation challenge with an Occlusion Metric (OM) score of 0.533. This was achieved through a combination of their proposed methods. Specifically, the table shows:
- An initial model trained on the "train" set achieved an OM of 0.514.
- Training on "train+val" and testing on "test" yielded an OM of 0.433.
- Extending training to 1200 epochs on "train+val" and testing on "test" improved the OM to 0.495.
- Finally, applying SWA for 12 additional epochs on top of the 1200-epoch training resulted in the winning score of 0.533.
Limitations and Future Work
- Dataset Size: While the augmentation helps, the inherent small size of the original dataset remains a challenge. The authors mention that their method is applicable to other data-scarce segmentation applications, suggesting its broader utility.
- Traditional CV for Augmentation: The reliance on traditional computer vision techniques (contour detection, Hough transform) for defining the "playable area" might be a bottleneck if these methods fail in challenging lighting or unusual court perspectives. Future work could explore end-to-end learned approaches for this.
- Generalizability Beyond Basketball: While the method is presented in the context of basketball, its effectiveness on other sports or general occlusion scenarios would be an interesting area for future research.
Summary updated