Learning To Recognize Occluded and Small Objects With Partial Inputs

Hasib Zunair, A. Ben HamzaWACV 2024

TL;DR

This paper introduces Masked Supervised Learning (MSL), a novel single-stage, model-agnostic framework for multi-label image recognition. MSL tackles the challenge of recognizing occluded and small objects by learning context-based representations through a masked input branch and modeling label co-occurrence with label consistency, achieving state-of-the-art performance.

What Problem Does This Paper Address?

Multi-label image recognition (MLIR) aims to identify multiple objects or attributes within a single image. This task is significantly complicated by occlusions (objects being partially hidden) and the presence of small objects, which are common in real-world scenarios like autonomous driving. Existing MLIR methods often struggle with these challenges. Many approaches rely on complex, multi-stage training pipelines, require large language models, or need extensive data augmentation and pretraining. Crucially, they often fail to explicitly address the problem of recognizing objects from partial or occluded views, leading to suboptimal performance. This paper aims to develop a simpler, more effective MLIR approach that can robustly handle occluded and small objects.

Methods

The core of MSL is a single-stage, model-agnostic training paradigm. It leverages a base recognition network (denoted as $f_\theta$) and introduces two key components:

  • Masked Branch (MaBr): This branch is trained on a heavily masked version of the input image ($I_{masked}$). By forcing the network to predict labels from partial inputs, it learns to extract context-based representations. This helps the model understand objects even when they are not fully visible. The masked image is created by applying random masks (with holes) to the original image.
  • Label Consistency (LaCo): To model the inherent co-occurrence of objects in images (e.g., a chair often appears with a table), MSL enforces consistency between the predictions of the main recognition branch ($y_p$) and the masked branch ($y_{mp}$). This is achieved by minimizing the L2-loss between their outputs, encouraging the model to learn relationships between visible and occluded object cues.

The overall loss function ($L_{total}$) combines the standard recognition loss ($L_{rcg}$), the masked branch loss ($L_{MaBr}$), and the label consistency loss ($L_{LaCo}$), with tunable hyperparameters to balance their contributions. The model-agnostic nature means MSL can be applied to various backbone architectures (e.g., CNNs, Transformers).

Datasets

The paper evaluates MSL on two standard multi-label image recognition benchmarks:

  • VOC2007: A widely-used dataset with 20 classes. The authors use the train-val set for training and the test set for evaluation.
  • MS-COCO (2014): A larger and more complex dataset with 80 classes, commonly used for various computer vision tasks.

Key Insights and Ideas

  • Occlusion as Partial Input: The fundamental insight is treating occluded objects as "partial inputs." This reframes the problem, suggesting that learning from incomplete visual information is key to robust recognition.
  • Masking for Context: Intuitively, masking parts of an image forces the model to rely on surrounding context to make predictions. This is a powerful way to learn richer, context-aware representations without explicit graph structures or complex relational modeling.
  • Simplicity and Model-Agnosticism: MSL's strength lies in its simplicity. It's a single-stage training process that doesn't require specialized architectures or complex data augmentation, making it broadly applicable and easier to implement.
  • Robustness to Masking: The experiments demonstrate that MSL is not only effective when trained with masked inputs but also robust to heavily masked inputs during testing, highlighting its ability to handle severe occlusions.

Key Results

  • State-of-the-Art Performance: MSL, when applied to various backbones (ResNet, ViT), achieves state-of-the-art results on both VOC2007 and MS-COCO benchmarks, particularly outperforming previous methods in terms of mAP. For instance, on VOC2007, MSL-C (using CSRA with ResNet-cut) achieved a mAP of 96.1%, a significant improvement.
  • Improved Recognition of Small/Occluded Objects: Qualitative results (Figure 2 and 4) clearly show MSL's superior ability to recognize small and occluded objects compared to baseline methods, which often fail in these scenarios.
  • Robustness Gains: When tested with randomly masked images, MSL showed substantial improvements in mAP, CR, and CF1 (e.g., 19.8% mAP improvement on VOC2007 with CSRA+ResNet-cut), demonstrating its resilience to partial inputs.
  • Efficiency: MSL requires only a single stage of training, unlike many multi-stage approaches, making it computationally more efficient.

Limitations and Future Work

  • Hyperparameter Tuning: While model-agnostic, MSL still requires tuning of the loss component hyperparameters ($\alpha_1, \alpha_2, \alpha_3$). The paper shows that $\alpha_3 = 0.5$ (for label consistency) yields the best results, suggesting its importance.
  • Future Work: The authors suggest adapting the MSL framework to other computer vision tasks, such as object detection, which also heavily benefits from understanding occluded and small objects. They also note the potential for further exploration of masking strategies within supervised learning.

Summary updated

BibTeX
@inproceedings{zunair2024learning,
  author = {Hasib Zunair and A. Ben Hamza},
  title = {{Learning To Recognize Occluded and Small Objects With Partial Inputs}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure