TL;DR
CROMPT is a new framework that adapts CLIP for semantic segmentation in adverse weather without ever seeing a target-domain image during training. By combining random visual noise prompts with a hybrid mix of learnable and hard-coded text, it achieves state-of-the-art zero-shot generalization across night, rain, fog, and snow.
What Problem Does This Paper Address?
Deep learning models for autonomous driving are notoriously brittle. A model trained on the sunny streets of Cityscapes often fails miserably when faced with a midnight downpour or a blinding snowstorm. This is a domain adaptation problem.
Standard Unsupervised Domain Adaptation (UDA) requires a large collection of unlabeled images from the target domain (e.g., "night") to bridge the gap. But what if you don't have those images? Zero-Shot Domain Adaptation (ZSDA) attempts to adapt models using only textual descriptions of the target environment.
While vision-language models like CLIP are great at this in theory, existing methods struggle with:
- Backbone Compatibility: Most prompt-learning methods are designed for Transformers, but many robust segmentation backbones still use ResNets.
- Pixel-Level Alignment: CLIP is trained on image-level descriptions, making it hard to align features for dense, pixel-level tasks like segmentation.
- Inference Constraints: Previous state-of-the-art methods (like PØDA) require knowing the "Domain ID" at test time to pick the right model—a luxury you don't always have in a moving vehicle.
Methods
The authors introduce CROMPT (Cross-Modal Prompt Learning for Domain Adaptation), which operates in two main stages: Zero-shot Feature Augmentation and Model Adaptation.
1. Cross-Modal Feature Augmentation (CROMA)
This is the core engine of the paper. It fine-tunes the CLIP vision encoder using two novel prompting strategies:
- Random Visual Prompting (RVP): Instead of learning complex visual prompts, the authors add fixed-size patches of non-learnable noise to the corners and center of the input image. Surprisingly, keeping the prompts as "noise" while making the CLIP vision encoder trainable allows the model to learn more robust, style-invariant features.
- Hybrid Textual Prompting (HTP): This combines "hard" prompts (static strings like "driving at night") with "soft" learnable prompts (context vectors optimized during training). This provides both stable semantic guidance and the flexibility to adapt to subtle domain shifts.
2. Adaptation Strategies
The authors propose two ways to deploy this:
- CROMA-VL: Uses a Vision-Language Model (Qwen-VL) to generate highly detailed, domain-conditioned descriptions of source images (e.g., describing exactly how a rainy city street looks) to guide the augmentation.
- CROMPT (Progressive Adaptation): A two-step "easy-to-hard" strategy. It first trains on "easy" domains (snow, fog) using hard prompts and "hard" domains (night, rain) using learnable prompts, then swaps them. This curriculum-style approach helps the model incrementally learn domain-invariant representations without needing a heavy VLM.
Datasets
The models were trained on Cityscapes (sunny/clear source domain) and evaluated on:
- ACDC: A benchmark specifically for adverse conditions, including Fog, Night, Rain, and Snow.
- GTA5: A synthetic dataset used to test generalization to "game" environments.
Key Insights and Ideas
- Noise as a Regularizer: The success of RVP suggests that adding structured noise to the input is an effective way to "force" the CLIP vision encoder to find more robust features during fine-tuning, rather than overfitting to specific visual prompts.
- Breaking the Domain ID Barrier: Unlike previous methods that needed to be told "it is raining" at inference time, CROMPT produces a single, unified model that generalizes across all weather conditions automatically.
- Joint Processing: By processing multiple domain prompts (Night, Rain, Fog, etc.) in a single forward pass, the model learns a shared latent space that is inherently more robust than training on each domain individually.
Key Results
CROMPT consistently outperforms strong baselines like PØDA and ULDA:
- mIoU Gains: Achieved an average improvement of 1.63%–2.63% mIoU over state-of-the-art methods on the ACDC dataset.
- Specific Conditions: The gains were particularly high in challenging scenarios, such as a 2.45% gain in source-to-rain adaptation.
- Efficiency: Crucially, all these gains come with zero additional computational cost at inference time, as the prompting mechanisms are only used during the training/adaptation phase.
Limitations and Future Work
The authors note that while the two-stage progressive adaptation is highly effective, extending it to a three-stage process actually led to a performance drop, suggesting there is a "sweet spot" for curriculum complexity.
Future work could explore how to better capture fine-grained details that VLMs might miss and how to scale this approach to even more diverse "open-world" scenarios beyond weather, such as different geographical architectures or rare road hazards.
Summary updated