A concept bottleneck model is a classifier forced to predict a short list of human-named quantities first, such as "temperature" or "season", and then to make its final prediction from those quantities alone, usually through a linear layer. The appeal is that an expert can read the weights and correct a wrong intermediate value at test time. The weakness is that the list is never complete. Recent fixes add learnable "residual" vectors to soak up whatever the named concepts miss. They then name those vectors by matching them to words. This note argues that the fixes repair accuracy while leaving the interpretation broken. When a real driver is missing, statistics has a precise name for what happens to the remaining weights: omitted variable bias, meaning the weights on the measured factors silently absorb the missing factor's effect. The orthogonality penalties that current residual methods use to keep residuals "clean" make this worse, because they lock the bias into the concept weights by construction.
We propose to treat the missing factor the way classical factor analysis does. A factor is identifiable because it leaves a common footprint on many outcomes. We place a concept bottleneck on top of a frozen Earth-observation foundation encoder. We train it to predict many land-cover outcomes at once, and we look for low-rank structure that is shared across the outcomes' residuals and absent from the named concepts. The test bed matters as much as the method. Ben-ge provides measured elevation, climate zone, reanalysis temperature and season for every Sentinel patch. We can therefore withhold a real physical variable, ask whether the shared residual recovers it, and ask whether the concept weights move back to where they would be had it been measured. Prior work never had ground truth for any of these questions.
Core Claim
With a single outcome, as in Incremental Residual Concept Bottleneck Models and Hybrid Concept Bottleneck Models, a residual vector cannot tell a missing physical driver apart from task-specific noise. Both are just "what the concepts did not explain." With $K$ related outcomes, a driver such as elevation shifts many class fractions together, so it shows up as a low-rank component of the cross-task residual. We claim three things. First, a rank-$r$ residual shared across tasks recovers withheld physical variables much better than single-task residuals, and recovery improves as $K$ grows. Second, refitting the linear heads on concepts plus the recovered factor, without an orthogonality constraint, reduces the bias in the concept weights. Orthogonalized residuals of the kind used by HybridCBM and by the Concept Orthogonal Loss of Auxiliary Losses for Learning Generalizable Concept-based Models leave that bias untouched. Third, the recovered factor yields a data-driven estimate of the sensitivity parameters in Omitted Variable Bias in Machine Learned Causal Models, which that framework otherwise leaves for the analyst to posit.
The second claim follows from the linear omitted-variable identity. Suppose outcome $y_k$ truly depends on concepts $c$ with weights $w_k$ and on a missing factor $u$ with weight $\delta_k$. Write $\Gamma$ for the coefficients from regressing $u$ on $c$. Then a model fit on $c$ alone learns
$$\tilde w_k = w_k + \Gamma\,\delta_k .$$
Any residual constrained to be uncorrelated with $c$ can only capture the part of $u$ orthogonal to $c$. That leaves the $\Gamma\delta_k$ term inside $\tilde w_k$, which is why orthogonality "freezes" the bias.
Research Program
The pipeline has four parts. The first is a frozen encoder: SkySense if its weights can be obtained (SkySense reports strong frozen-backbone transfer). Any other frozen remote-sensing encoder would also serve, and an ImageNet encoder is included as a domain-gap control. The second is a concept layer. It predicts from the embedding the measured variables we keep, and is supervised by their measured values, so test-time intervention simply substitutes the true measurement. The third is a rank-$r$ residual map from the embedding. The fourth is one linear head per outcome, reading both concepts and residual.
The residual is fit in the simplest way that captures the identification idea. We first fit concept-only heads. We then run reduced-rank regression of the stacked $K$ outcome residuals on the embedding, which finds the $r$ embedding directions that best predict all residuals jointly. Finally, we refit all heads on concepts plus those $r$ directions with no orthogonality penalty. A jointly trained version with the same structure is an ablation, not the main method. We deliberately keep the residual low-rank and shared, and we skip text decoding. The residual is interpreted by grounding against measured rasters. A variable withheld during training is available for evaluation only.
The outcomes come from Ben-ge's labels: per-patch class fractions for each ESA WorldCover class, derived from its segmentation masks, together with the CORINE multi-label targets inherited from BigEarthNet. In each experiment, one physical variable is withheld from the concept set and the others are kept as concepts. The withheld variables are elevation from the Copernicus DEM, Köppen-Geiger climate zone, and ERA-5 temperature. Elevation is the primary case, because it is partly visible in imagery through snow, shading and vegetation, yet is not itself a land-cover label.
The program has an analysis arm. We use the recovered factor to estimate how much of each outcome's residual variance it explains and how strongly it is predicted by the concepts. We plug those partial $R^2$ values into the Chernozhukov et al. bias bound and check whether the bound computed this way covers the true weight shift we observe when the withheld variable is restored. This turns their sensitivity analysis from a "what if" exercise into a calibrated check.
Baselines
The naive reference point is a concept-only bottleneck on the frozen embedding, following Concept Bottleneck Models: concepts predicted from the embedding, a linear head per outcome, and no residual. It sets the accuracy floor and the biased weights $\tilde w_k$. The accuracy ceiling is an unconstrained linear probe on the same frozen embedding. It has no interpretability, but tells us how much accuracy the bottleneck costs. The interpretable oracle is the same bottleneck with the withheld variable restored as a concept. Its weights define "unbiased" for our metrics, and its accuracy defines what a perfect recovery would achieve. For context on the input-fusion route, Ben-ge reports classification F1 of 85.12%±0.34 with all modalities as model inputs, 82.81%±0.29 with only Sentinel-2 and climate, and best segmentation IoU of 39.71%±0.21. These numbers come from supervised black-box networks, not frozen-probe bottlenecks, so they bound the problem rather than serve as directly comparable baselines.
The strongest existing competitors are the residual concept methods. One is a single-task residual in the style of Res-CBM. The other is a HybridCBM-style dynamic bank with orthogonality and distribution-alignment losses. Both are run with one model per outcome, on the same embeddings, splits, and hyperparameter search budget, and with the same total residual dimension as our rank-$r$ shared residual, so that capacity is matched. On standard vision benchmarks, HybridCBM reaches 68.38% on CIFAR-100 against 70.1% for a CLIP linear probe (Hybrid Concept Bottleneck Models). Res-CBM improves on PCBM by 13.41% on CIFAR-100 (Incremental Residual Concept Bottleneck Models). No reported result was found on Lacuna for either method on Ben-ge or any Earth-observation benchmark, so we will produce those numbers ourselves.
Beating these competitors means the following. Our method should recover the withheld variable with clearly higher canonical correlation on held-out patches. It should cut the distance between learned and oracle concept weights. And it should do both at equal or better outcome accuracy, all across three random splits.
Three ablations isolate the key ingredient. The first sets $K=1$, which removes the multi-task footprint. The second runs our method with an orthogonality constraint added, testing whether orthogonality preserves the bias. The third replaces the shared low-rank residual with independent per-task residuals of the same total size.
Why not just…?
Why not just use an unconstrained probe on the frozen features? It will probably match or beat us on accuracy, and we report that gap honestly. However, it offers no weights an expert can read and no intervention handle. The CUB+Places results in Concept Bottleneck Models show standard models degrading from 17.5% to 62.7% error under background shift, against roughly 48–49% for bottlenecks. Our question is about the correctness of the readable weights, which a probe does not have.
Why not just add the missing variable as an input, as Ben-ge does? When the variable can be measured, one should. Ben-ge's gains from fusing DEM and climate show it matters. The method targets drivers that are not measured at deployment scale, such as soil moisture deficit or nitrogen limitation in the vegetation studies by Cheng et al. and Yin et al.. Withholding Ben-ge variables is how we validate it against ground truth before such a use.
Why not just run post-hoc PCA on the stacked residuals? This is a real threat. It is close to our main estimator, and that closeness is why we chose the reduced-rank version over heavier machinery. What PCA of residuals lacks is the tie to the embedding: it describes residuals but gives no map from a new image to the factor. That map is needed for prediction and for refitting the heads. We include residual-PCA projected back onto the embedding as a baseline. If it does as well, the contribution narrows to the identification and bias findings, which we would state plainly.
Why not just use DML sensitivity analysis? Omitted Variable Bias in Machine Learned Causal Models bounds a causal quantity under a hypothesized confounder strength. It does not say what the confounder is. In their gasoline-demand example, a confounder explaining about 2% of variance sufficed to erase the estimate. We use their bound as an evaluation instrument, not as a competitor.
Why not just balance task gradients with RotoGrad? RotoGrad addresses conflicting gradients in a shared deep backbone. Here the backbone is frozen, and the main estimator is fit in closed form, so there is little gradient conflict to resolve. We apply magnitude balancing only in the jointly trained ablation, where classification and fraction-regression losses differ in scale.
Proposed Experiments
The decisive check is a recovery curve. For each withheld variable, we plot the canonical correlation between the recovered factor and the true raster against the number of outcomes $K$, using random subsets of the WorldCover and CORINE targets. Flat curves would falsify the core claim. Curves that rise with $K$, and sit above the single-task Res-CBM-style and HybridCBM-style residuals at $K=1$, would support it.
The second check is weight repair. We measure the relative distance between learned and oracle concept weights for the concept-only model, the orthogonalized residual, and our unconstrained refit. The identity above predicts that the orthogonalized version stays near the concept-only value. Our refit should move toward the oracle in proportion to how well the factor was recovered.
The third check is intervention fidelity. We substitute measured concept values at test time and compare the resulting output changes with those of the oracle model.
The fourth check is calibration. We ask whether partial-$R^2$ bounds computed from the recovered factor cover the observed weight shifts.
A domain-gap control repeats the elevation experiment with an ImageNet encoder. The BigEarthNet results, 48.23% precision for ImageNet Inception-v2 against 69.93% for a multispectral CNN (Bigearthnet), suggest that recovery depends on what the frozen space encodes. This control separates failures of the method from failures of the features.
References
- Incremental Residual Concept Bottleneck Models
- Hybrid Concept Bottleneck Models
- Auxiliary Losses for Learning Generalizable Concept-based Models
- Long Story Short: Omitted Variable Bias in Causal Machine Learning
- SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery
- Concept Bottleneck Models
- Ben-Ge: Extending Bigearthnet with Geographical and Environmental Data
- Effects of Climate Change on Temperature Sensitivity of Vegetation Growth in Huang-Huai-Hai Plain: Spatial-Temporal Dynamics and Ecological Adaptability
- Spatial-Temporal Evolution of Vegetation NDVI in Association with Climatic, Environmental and Anthropogenic Factors in the Loess Plateau, China during 2000-2015: Quantitative Analysis Based on Geographical Detector Model
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- BigEarthNet: A Large-Scale Benchmark Archive For Remote Sensing Image Understanding