Occlusion Robustness of CLIP for Military Vehicle Classification

Jan Erik van Woerden, Gertjan J. Burghouts, Lotte Nijskens, Alma M. Liezenga, Sabina van Rooij, Frank Ruis, Hugo J. Kuijf2025 · arXiv

Abstract

Contrastive Language-Image Pre-training (CLIP) models have demonstrated remarkable zero-shot capabilities across diverse computer vision tasks. However, their reliability in high-stakes military environments—where targets are frequently obscured by terrain, weather, or structures—remains under-explored. This study evaluates the robustness of various CLIP architectures against synthetic occlusions, ranging from contiguous blocks to dispersed noise like rain and snow. Our findings indicate that Vision Transformers (ViTs) significantly outperform Convolutional Neural Networks (CNNs) in maintaining performance under occlusion. Furthermore, we identify a "fine-tuning paradox" where standard adaptation techniques improve clean accuracy at the expense of environmental resilience.

Introduction

In tactical reconnaissance and surveillance, the ideal "clean" image is a rarity. Military vehicles are often partially hidden by foliage, buildings, or camouflaging nets (contiguous occlusion), or their visual signature is degraded by atmospheric conditions such as smoke, dust, and precipitation (dispersed occlusion). While Vision-Language Models (VLMs) like CLIP offer a solution to the data-scarcity problem in defense—allowing for classification without massive labeled datasets—their training on curated internet imagery may not prepare them for the "dirty" reality of the battlefield.

This research systematically investigates how different CLIP variants handle these challenges, focusing on the trade-offs between model architecture, scale, and fine-tuning strategies.

Methodology

Evaluation Metrics

To quantify performance, the researchers utilized two primary metrics:

  1. Top-1 Accuracy: The model's precision on unobstructed, high-quality images.
  2. Normalized Area Under the Curve (NAUC): A robustness metric that measures how gracefully a model's performance degrades as the level of occlusion increases from 0% to 100%.

Occlusion Types

The study simulated five distinct types of physical and environmental interference:

  • Contiguous Occlusions: "Slide blackout" (simulating a vehicle emerging from cover) and "Bar occlusion" (simulating viewing through fences or branches).
  • Dispersed Occlusions: "Random rain," "Random snow," and "Grid dropout" (simulating sensor degradation or fine-grained foliage).

Models and Data

Nine CLIP variants were evaluated, spanning different architectures (CNN vs. ViT) and scales (90M to 1.88B parameters). These included the original CLIP, CLIPA, and Perception Encoder-Core (PE-Core) models. The evaluation was conducted on a specialized dataset of 18 military vehicle classes—including the T-90 tank, Leopard 2, BTR-80 APC, and M109 self-propelled artillery—compiled from public sources with approximately 50 images per class.

Key Insights and Results

The Superiority of Global Attention

A primary finding of the study is the significant robustness gap between architectures. Vision Transformers (ViTs) consistently outperformed ResNet-based CNNs. For instance, the ViT-B/16 model achieved a 22 percentage point higher NAUC than ResNet50, despite having a similar parameter count. This is attributed to the ViT's global self-attention mechanism, which allows the model to integrate information from across the entire image. While a CNN may fail if its local receptive field is blocked, a ViT can "look around" an occlusion to identify a vehicle based on distant, uncorrupted features.

The Vulnerability to Dispersed Noise

The research highlights that dispersed occlusions (rain and snow) are far more damaging than contiguous blocks. While a model might still recognize a tank if the turret is visible but the hull is blocked, dispersed noise "poisons" the feature extraction process globally. This causes the model to lose focus on the target and attend to the background noise, leading to a rapid collapse in classification accuracy.

The Fine-Tuning Paradox

One of the most critical discoveries involves the impact of model adaptation. Linear probing—a common technique where only the final classification head is trained—was found to be a double-edged sword. While it significantly boosted accuracy on clean images (e.g., from 55.6% to 84.0% for CLIPA-ViT-L), it caused a "robustness crash" under dispersed occlusion, with the NAUC for snow dropping from 54.0% to 36.0%. This suggests that linear probing forces the model to overfit to specific high-level features that are easily obscured by environmental noise.

Full fine-tuning (updating all weights) proved more stable, reaching a peak clean accuracy of 93.3% and mitigating some of the robustness loss, though it still struggled to match the consistency of the original zero-shot model.

Top Performer

The PE-Core-ViT-G/14-448 (1.88B parameters) emerged as the most resilient model. Benefiting from both massive scale and a pre-training "recipe" that includes heavy data augmentation (cropping and color jittering), it achieved an average NAUC of 76.6% across all tested occlusion types.

Conclusion and Future Directions

While CLIP models show great promise for military applications, their deployment in unconstrained environments requires careful consideration of architecture and training methodology. Transformers are inherently more robust to physical obstructions than CNNs, but the process of fine-tuning them for specific tasks can inadvertently strip away their natural resilience to weather and noise.

Future research should focus on "baking in" robustness through occlusion-specific augmentations during pre-training and exploring masking-based objectives (similar to Masked Autoencoders). By forcing models to reconstruct missing or obscured image regions during training, researchers can develop VLMs that are truly ready for the tactical realities of the field.

Summary updated

Abstract

Vision-language models (VLMs) like CLIP enable zero-shot classification by aligning images and text in a shared embedding space, offering advantages for defense applications where labeled data is scarce. However, CLIP is primarily trained on high-quality internet imagery. Its robustness in challenging military operational environments, characterized by factors like partial occlusion and degraded signal-to-noise ratio (SNR) because of obscurant objects or weather, remains underexplored. We investigate the robustness of several CLIP variants to occlusion, using a custom dataset of 18 military vehicle classes. We simulate both contiguous occlusions (slide blackout, bar occlusion) and dispersed occlusions (random rain, snow, grid dropout) to reflect real-world environmental challenges. Robustness is evaluated using Normalized Area Under the Curve (NAUC) across occlusion percentages. Four key insights emerge: (1) Fine-grained, dispersed occlusions (e.g., snow, rain) degrade performance more than larger contiguous occlusions—NAUC of 61.3% for dispersed occlusion vs 78.9% for contiguous occlusions for PE-Core-ViTL/ 14-336 (2) Transformer-based CLIP models consistently outperform CNN-based CLIP models, with ViT-B/16 achieving up to 22 percentage point higher NAUC than ResNet50; (3) Pre-training methodology significantly affects robustness— PE-Core models consistently outperform CLIPA counterparts at similar scales (e.g., +6.7pp NAUC at ~320M parameters), showing that improved pre-training augments robustness beyond scaling alone; (4) Fine-tuning introduces a trade-off— linear probing boosts clean-image accuracy (55.6%→88.0%) but reduces robustness under dispersed occlusions (snow NAUC 54.0%→36.0%), while full fine-tuning mitigates this effect (snow NAUC 44.5%) yet still falls short of zero-shot consistency. These results underscore the importance of occlusion-specific augmentations during training and the need for further exploration into patch-level sensitivity and architectural resilience for real-world deployment of CLIP.
BibTeX
@misc{woerden2025occlusion,
  author = {Jan Erik van Woerden and Gertjan J. Burghouts and Lotte Nijskens and Alma M. Liezenga and Sabina van Rooij and Frank Ruis and Hugo J. Kuijf},
  title = {{Occlusion Robustness of CLIP for Military Vehicle Classification}},
  year = {2025},
  eprint = {2508.20760},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.20760},
}
Figure