Weakly supervised semantic segmentation trains a pixel-level segmenter from image-level tags such as "this photo contains a car." The usual route goes through class activation maps, or CAMs, which are heatmaps showing where a trained classifier found evidence for each class. These maps are known to cover only the most telling part of an object, the bird's head or the car's wheels. This failure is called discriminative region bias. Every fix in the literature assumes the bias comes from semantics, meaning some parts are simply more informative than others. Real outdoor imagery adds a second cause. Non-homogeneous haze, meaning fog, smoke, or mist whose thickness varies patch by patch, hides some parts of an object and leaves others visible. Our hypothesis is that under patchy haze the classifier's "discriminative region" becomes mostly the least-hazed region. If so, the tools built to expand CAMs, such as erasing the top region or forcing image tiles to classify on their own, fail in predictable ways.
We propose to measure this visibility-driven bias directly, and then to remove it with one mechanism. An estimated haze-transmission map (how much scene light survives the haze at each pixel) is turned into a soft, additive bias on the self-attention scores of a vision transformer. The bias is applied only in its middle layers, and the transformer keeps features at 1/8 input resolution throughout. The bias stops tokens buried in dense haze from writing their washed-out features into clear tokens. Clear tokens can still pass information into hazed ones. The outer layers keep global attention so the model can still estimate scene-wide atmospheric light. The contribution is a diagnostic plus a single architectural change, tested against strong simpler alternatives that might make it unnecessary.
Core Claim
The first claim is empirical. On paired clean and hazy versions of the same image, CAM mass inside the true object mask shifts toward high-transmission pixels. The size of this shift is measurable, and it is not explained by object part.
The second claim is methodological. Let $q_i$ and $k_j$ be the query and key of tokens $i$ and $j$, let $d$ be the key dimension, and let $\hat t_j \in (0,1]$ be the estimated transmission at token $j$. For the middle band of layers we replace the attention score with
$$s_{ij} = \frac{q_i k_j^\top}{\sqrt d} + \lambda \log \hat t_j .$$
Here $\lambda \ge 0$ is a learned scalar per layer. This is equivalent to multiplying each key's unnormalized attention weight by $\hat t_j^{\lambda}$. As $\lambda$ grows, the bias approaches hard $-\infty$ masking of fully opaque keys. The claim is that this bias reduces the visibility skew and improves pseudo-label and final segmentation mIoU under haze. It should do so more than three alternatives: haze-augmented training alone, dehazing the image first, or feeding the density map in as an extra input.
How This Relates to Prior Work
Several methods attack discriminative region bias on clean images, and they motivate the problem. Adversarial erasing (Kweon et al.) hides the top region to force discovery of others. Puzzle-CAM makes 2×2 tiles reproduce the full-image CAM. SEAM enforces consistency of CAMs under affine transforms. ToCo counters the tendency of transformer patch tokens to become indistinguishable from one another in deep layers. All four treat bias as semantic. Under patchy haze, erasing the one clear region or classifying an opaque tile removes the only usable evidence. We keep ToCo as the base learner and change only how attention flows.
DiG is the closest work that joins CAMs and attention. It cross-attends from a ViT into frozen diffusion-model features to sharpen CAMs, and it is the strongest clean-image result in our file. DiG does not mask attention. Its affinity consistency also assumes a clean image plus mild added noise, an assumption that patchy haze breaks.
For the masking mechanism we borrow from layout-controlled image generation, which supplies the method. Cross Attention Control and Freestyle Layout-to-Image Synthesis clamp attention outside user-given masks. Seg2Any found that isolating attention only in middle layers prevents attribute leakage while preserving global lighting. That finding constrains our design, and it is why outer layers stay global. Those works use crisp, user-supplied binary masks. Our mask is continuous, estimated from the image, and asymmetric: only the key side is biased.
Dehazing work supplies the density signal and also warns where it fails. PMDNet estimates a per-pixel density encoding without density labels. FFA-Net uses sigmoid pixel attention to put more capacity on hazy pixels. RPD-Diff gives more diffusion steps to dense patches. All three output RGB images, not semantics. PMDNet also documents the confusion between bright surfaces and airlight, which bounds how much an estimated $\hat t$ can be trusted.
The resolution choice follows UViT, which keeps a constant 1/8 feature map with window-masked early attention. It also follows thin-object selection work, which shows that 1-pixel structures vanish under downsampling. The file also names a tension here: high-resolution streams carry haze noise into deep layers. The attention bias is our proposed answer, because it filters that noise at the level of messages between tokens rather than by downsampling. Finally, ACDC and WeatherProof show that fog degrades fully supervised segmenters. Neither studies weak supervision.
Research Program
The first step is the data, since no hazy weakly supervised benchmark exists. We build Hazy-VOC by rendering non-homogeneous haze onto PASCAL VOC 2012. Transmission is $t(x)=e^{-\beta(x) d(x)}$, where $d$ is the output of a monocular depth estimator. The scattering field $\beta(x)$ is spatially varying procedural noise, with amplitude and patch-size statistics fitted to the 55 smoke-machine pairs of NH-Haze. Every hazy image keeps its clean twin and its true $t$. For real fog we use the ACDC fog split. Image-level tags come from its panoptic labels, restricted to object classes (car, person, bus, pole, sign), and its invalid masks are excluded from scoring.
The second step is the diagnostic. For each image pair we compute a visibility skew. It is the ratio of mean CAM inside the ground-truth mask on the top-transmission third of object pixels to that on the bottom third. We divide this by the same ratio computed on the clean twin, using the same pixel partition, so that differences in part saliency cancel out. A normalized skew well above one means visibility, not semantics, is choosing the discriminative region.
The third step is the model. It is ToCo's training recipe on a single-scale ViT at 1/8 resolution, with UViT-style windowed early layers. The transmission bias is added to a middle band of layers. $\hat t$ comes either from a frozen PMDNet-style density estimator or, as an oracle, from the true synthetic $t$. We sweep which layer band receives the bias and whether the bias is soft (learned $\lambda$) or hard ($-\infty$ below a threshold).
Baselines
The naive reference is ToCo trained and evaluated on Hazy-VOC with no haze-specific change. Because its training images are hazy, it is already a haze-augmentation baseline, and it is the floor any method must clear. For context we also report ToCo on clean VOC. That paper reports 72.2% test mIoU (ToCo), and the gap from this number measures the cost of haze.
The strongest existing method is DiG, which reports 73.9% mIoU on PASCAL VOC 2012 val and 45.5% on MS-COCO 2014 (DiG). There is no reported result on Lacuna for any weakly supervised method on hazy VOC or on ACDC fog with image-level labels, so we will rerun DiG on Hazy-VOC ourselves. DiG is the toughest competitor because its diffusion features might already be robust to haze.
The dehaze-first baseline runs PMDNet and FFA-Net as frozen preprocessors in front of ToCo. Each preprocessor is fine-tuned on matching synthetic haze so it is not handicapped by a domain gap. The density-as-input baseline concatenates $\hat t$ to the RGB input and adds FFA-style sigmoid pixel gating. It gets the same density estimate as our model, so any gain we show cannot come from extra information.
All methods share the same backbone size where applicable, the same training images and schedule, the same pseudo-label to DeepLab retraining stage, and a matched hyperparameter search of 20 trials each. Beating the field means two things on Hazy-VOC val: higher pseudo-label mIoU and higher final mIoU than DiG and dehaze-first, with a margin above the spread across three seeds, together with a lower normalized visibility skew.
The key ablations remove one ingredient at a time. Bias in all layers versus the middle band tests whether atmospheric context needs global outer layers. The asymmetric key-side bias versus a symmetric query-and-key bias tests whether the one-way information flow matters. Estimated versus oracle $\hat t$ bounds the loss from imperfect density estimates. 1/16 versus 1/8 resolution tests whether high-resolution features help or hurt once the bias is present.
Why not just…?
Why not just train on hazy images and let the network adapt? This is the most serious threat, and it is exactly our naive baseline. If ToCo trained on Hazy-VOC matches our model's mIoU and shows normalized skew near one, the architectural claim is false and we will report it that way. The reason to expect a gap is that data alone gives the classifier no incentive to go beyond the clear region. Classification loss already drops near zero once one visible part is found, which is the mechanism behind discriminative bias itself.
Why not just dehaze first and then segment? Restoration of opaque patches must hallucinate texture. PMDNet documents that bright surfaces get confused with fog, and the NTIRE report notes that hallucination is unavoidable in dense regions. Both errors can create false evidence for the classifier. Still, a strong dehazer could win on light haze. We therefore report results stratified by haze density, and we expect dehaze-first to be competitive at low density and worse at high density. If it wins everywhere, the attention bias is not worth its complexity.
Why not just give the network the density map as an input or a soft gate? A soft gate rescales pixels, but it does not stop hazed tokens from mixing into clear ones through attention. The generation literature (Freestyle Layout-to-Image Synthesis, Seg2Any) found that this kind of residual mixing is what causes leakage. The density-as-input baseline receives identical $\hat t$, so the comparison isolates where the density signal enters the network rather than whether it is available.
Decisive Checks
Three outcomes would decide the proposal. First, the diagnostic must show a normalized visibility skew clearly above one for ToCo and DiG on Hazy-VOC. If it does not, the premise fails and the method is unmotivated. Second, the gain of the middle-band bias over the haze-trained and density-as-input baselines must grow with haze density and hold with estimated, not just oracle, $\hat t$. Third, on ACDC fog, the model must improve IoU on thin classes such as pole and sign at 1/8 resolution without losing on large classes. This would show that the bias resolves the conflict between keeping high-resolution features and propagating haze noise. If it holds only on synthetic haze, we will report it as a synthetic-only result.
References
- Unlocking the Potential of Ordinary Classifier: Class-specific Adversarial Erasing Framework for Weakly Supervised Semantic Segmentation
- Puzzle-CAM: Improved Localization Via Matching Partial And Full Features
- Self-Supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation
- Token Contrast for Weakly-Supervised Semantic Segmentation
- Diffusion-Guided Weakly Supervised Semantic Segmentation
- Localized Text-to-Image Generation For Free via Cross Attention Control
- Freestyle Layout-to-Image Synthesis
- Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Control
- Perceiving and Modeling Density for Image Dehazing
- FFA-Net: Feature Fusion Attention Network for Single Image Dehazing
- RPD-Diff: Region-Adaptive Physics-Guided Diffusion Model for Visibility Enhancement under Dense and Non-Uniform Haze
- A Simple Single-Scale Vision Transformer for Object Detection and Instance Segmentation
- Deep Interactive Thin Object Selection
- ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding
- WeatherProof: A Paired-Dataset Approach to Semantic Segmentation in Adverse Weather
- NTIRE 2020 Challenge on NonHomogeneous Dehazing