Abstract
Contrastive Language-Image Pre-training (CLIP) models have demonstrated remarkable zero-shot capabilities across diverse computer vision tasks. However, their reliability in high-stakes military environments—where targets are frequently obscured by terrain, weather, or structures—remains under-explored. This study evaluates the robustness of various CLIP architectures against synthetic occlusions, ranging from contiguous blocks to dispersed noise like rain and snow. Our findings indicate that Vision Transformers (ViTs) significantly outperform Convolutional Neural Networks (CNNs) in maintaining performance under occlusion. Furthermore, we identify a "fine-tuning paradox" where standard adaptation techniques improve clean accuracy at the expense of environmental resilience.
Introduction
In tactical reconnaissance and surveillance, the ideal "clean" image is a rarity. Military vehicles are often partially hidden by foliage, buildings, or camouflaging nets (contiguous occlusion), or their visual signature is degraded by atmospheric conditions such as smoke, dust, and precipitation (dispersed occlusion). While Vision-Language Models (VLMs) like CLIP offer a solution to the data-scarcity problem in defense—allowing for classification without massive labeled datasets—their training on curated internet imagery may not prepare them for the "dirty" reality of the battlefield.
This research systematically investigates how different CLIP variants handle these challenges, focusing on the trade-offs between model architecture, scale, and fine-tuning strategies.
Methodology
Evaluation Metrics
To quantify performance, the researchers utilized two primary metrics:
- Top-1 Accuracy: The model's precision on unobstructed, high-quality images.
- Normalized Area Under the Curve (NAUC): A robustness metric that measures how gracefully a model's performance degrades as the level of occlusion increases from 0% to 100%.
Occlusion Types
The study simulated five distinct types of physical and environmental interference:
- Contiguous Occlusions: "Slide blackout" (simulating a vehicle emerging from cover) and "Bar occlusion" (simulating viewing through fences or branches).
- Dispersed Occlusions: "Random rain," "Random snow," and "Grid dropout" (simulating sensor degradation or fine-grained foliage).
Models and Data
Nine CLIP variants were evaluated, spanning different architectures (CNN vs. ViT) and scales (90M to 1.88B parameters). These included the original CLIP, CLIPA, and Perception Encoder-Core (PE-Core) models. The evaluation was conducted on a specialized dataset of 18 military vehicle classes—including the T-90 tank, Leopard 2, BTR-80 APC, and M109 self-propelled artillery—compiled from public sources with approximately 50 images per class.
Key Insights and Results
The Superiority of Global Attention
A primary finding of the study is the significant robustness gap between architectures. Vision Transformers (ViTs) consistently outperformed ResNet-based CNNs. For instance, the ViT-B/16 model achieved a 22 percentage point higher NAUC than ResNet50, despite having a similar parameter count. This is attributed to the ViT's global self-attention mechanism, which allows the model to integrate information from across the entire image. While a CNN may fail if its local receptive field is blocked, a ViT can "look around" an occlusion to identify a vehicle based on distant, uncorrupted features.
The Vulnerability to Dispersed Noise
The research highlights that dispersed occlusions (rain and snow) are far more damaging than contiguous blocks. While a model might still recognize a tank if the turret is visible but the hull is blocked, dispersed noise "poisons" the feature extraction process globally. This causes the model to lose focus on the target and attend to the background noise, leading to a rapid collapse in classification accuracy.
The Fine-Tuning Paradox
One of the most critical discoveries involves the impact of model adaptation. Linear probing—a common technique where only the final classification head is trained—was found to be a double-edged sword. While it significantly boosted accuracy on clean images (e.g., from 55.6% to 84.0% for CLIPA-ViT-L), it caused a "robustness crash" under dispersed occlusion, with the NAUC for snow dropping from 54.0% to 36.0%. This suggests that linear probing forces the model to overfit to specific high-level features that are easily obscured by environmental noise.
Full fine-tuning (updating all weights) proved more stable, reaching a peak clean accuracy of 93.3% and mitigating some of the robustness loss, though it still struggled to match the consistency of the original zero-shot model.
Top Performer
The PE-Core-ViT-G/14-448 (1.88B parameters) emerged as the most resilient model. Benefiting from both massive scale and a pre-training "recipe" that includes heavy data augmentation (cropping and color jittering), it achieved an average NAUC of 76.6% across all tested occlusion types.
Conclusion and Future Directions
While CLIP models show great promise for military applications, their deployment in unconstrained environments requires careful consideration of architecture and training methodology. Transformers are inherently more robust to physical obstructions than CNNs, but the process of fine-tuning them for specific tasks can inadvertently strip away their natural resilience to weather and noise.
Future research should focus on "baking in" robustness through occlusion-specific augmentations during pre-training and exploring masking-based objectives (similar to Masked Autoencoders). By forcing models to reconstruct missing or obscured image regions during training, researchers can develop VLMs that are truly ready for the tactical realities of the field.
Summary updated