Counting Parts to Read Function: Compositional Multiple Instance Learning for Land Use

Satellite models are good at saying what a place is made of, such as roofs, asphalt or grass. They are much worse at saying what a place is for, such as a school, a warehouse or a mall. One reason is that function is rarely written in any single object. A parking lot sits next to a mall and also next to a park, and a flat grey roof covers both a warehouse and a retail center. What separates them is the combination: how many loading docks there are, how much of the parcel is paved, whether a running track sits beside a mid-sized building. This note proposes to treat a land parcel as a bag of small image regions. Each region is assigned to a learned vocabulary of physical "part types," and the bag's function is predicted from the counts of those part types using gradient-boosted decision trees. Trees express rules like "at least three docks and more than 40% pavement" natively. The image encoder that produces the regions uses importance-weighted downsampling, so that parts only a few pixels wide survive long enough to be counted.

The claim is specific and testable. Standard multiple instance learning (MIL), which learns from labels attached to whole collections of image patches rather than to individual patches, assumes that a bag is positive if some witness patch is positive. That assumption is wrong for land function. The confusable classes differ in the composition of shared parts, not in the presence of a unique part. A threshold-and-conjunction aggregator over part counts should therefore beat attention pooling on exactly the confusable class pairs. The gain should depend on preserving small parts, so it should grow as ground resolution gets finer and vanish when the encoder discards small structures.

What the literature establishes and leaves open

Four lines of evidence motivate the gap. Urban Building Classification (UBC) labels individual buildings in Beijing and Munich and finds instance-segmentation models reach AP 22.8 on roof type but only 14.0 on building function on the same imagery. Appearance of a single building does not carry function. Using Convolutional Networks and Satellite Imagery to Identify Patterns in Urban Environments at a Large Scale classifies European urban zones and finds small crops fail without neighborhood context. Large tiles, however, blur the small structures that carry the signal. Functional Map of the World (fMoW) builds a 63-class, million-image functional benchmark and shows that image-only DenseNet-161 reaches average F1 0.679, that adding sensor and location metadata lifts it to 0.722, and that a full temporal ensemble reaches 0.734. Non-visual tabular signals clearly matter, but fMoW combines them through neural heads. Fine-Grained Land Use Classification at the City Scale Using Ground-Level Images frames the same problem as the limit of the view from above and turns to street imagery, which covers commercial corridors well and industrial or rural land poorly.

The closest existing method is MILRDA. It applies deep MIL with channel-attention pooling to remote-sensing scene classification and reports 91.56% overall accuracy on NWPU-RESISC45 with 0.66M parameters. It works on pre-cropped scene tiles, aggregates with soft attention, and does not target confusable functional pairs. Our proposal differs in its aggregation assumption: compositional counts read by trees, not a weighted average dominated by salient patches. It also uses parcel or footprint bags rather than fixed tiles.

Two papers supply methods we reuse. LIP: Local Importance-Based Pooling replaces fixed pooling with a learned importance map before subsampling, raising COCO small-object AP from 21.9 to 24.0 in Faster R-CNN. We put it inside the patch encoder of the MIL pipeline, where it has never been used. DOFEN makes oblivious decision-tree ensembles differentiable and matches XGBoost and CatBoost on tabular benchmarks. We use it only in a secondary variant that lets gradients from the tree aggregator reach the part vocabulary.

Two papers constrain the design. A Multi-Branch Feature Fusion Network for Building Detection in Remote Sensing Images shows high-resolution local crops help small buildings on xView (36.54% AP versus 34.26% from the global downsampled view). It also documents the memory cost of many crops, so our bags must stay small. The vision-language reasoner in Urban Socio-Semantic Segmentation with Vision-Language Reasoning handles function well on SocioSeg but costs 2.71 s per sample and tends to find one dominant instance per scene. That is the opposite of the many-part parcels we target, and it sets the efficiency bar we aim to beat by orders of magnitude.

Core Claim

For a bag $B$ of $K$ patches, a shared encoder with importance pooling maps each patch to an embedding $h_k$. A codebook of $M$ learned part prototypes $c_1,\dots,c_M$ softly assigns patches to part types. The bag is then summarized by a count vector plus area-weighted fractions:

$$n_m(B) = \sum_{k=1}^{K} \frac{\exp(-\lVert h_k - c_m\rVert^2/\tau)}{\sum_{j}\exp(-\lVert h_k - c_j\rVert^2/\tau)}, \qquad m = 1,\dots,M,$$

where $\tau$ is a temperature controlling assignment sharpness. The final feature vector concatenates $n(B)$, $n(B)/K$, bag size $K$, and the fMoW-style metadata (ground sample distance, location zone, sun and off-nadir angles). A gradient-boosted tree ensemble predicts the bag's function from this vector.

The hypothesis is that tree splits on these counts capture conjunctions of thresholds, for example "many small bright rooftop units and little green space." Attention pooling over embeddings tends to average such conjunctions away, and it tends to over-weight co-occurring background such as parking. The research file lists this attention failure under contextual confounding as an open MIL problem.

Research Program

The first stage builds bags. On fMoW, each labeled bounding box together with a context ring becomes a bag of fixed-size, full-resolution patches. On UBC, we group buildings into blocks by spatial clustering within tiles and give each block its majority functional label. Training then uses only block labels, and the per-building labels are held out for evaluating localization.

The second stage trains the encoder and codebook with a bag-level classification loss through a lightweight attention MIL head, which serves only as a training scaffold. The codebook prototypes are initialized by k-means. The third stage freezes the encoder, computes count vectors, and fits gradient-boosted trees.

A secondary variant replaces the trees with a DOFEN ensemble trained end-to-end. This tests whether gradients from a tree-shaped aggregator produce a better part vocabulary than gradients from attention. The analysis then asks which part types the trees split on, whether those types correspond to small structures, and whether that pattern holds for the confusable class pairs.

Baselines

The naive reference point is a mean-pooled bag embedding from the same encoder, followed by a linear classifier. It shows how much any structured aggregation adds at all.

The strongest reported fMoW comparator is the paper's own metadata-fused model. Functional Map of the World reports average F1 0.722 for CNN plus metadata and 0.734 for its full temporal ensemble. We will report against both on the official split. Our main comparison uses single-time imagery plus metadata, so 0.722 is the matched number. To beat it, the model must exceed 0.722 average F1 without temporal input, and must show a larger margin on the confusable class pairs we predefine from the paper's confusion structure.

The strongest method-level competitor is attention-based MIL with metadata concatenated to the pooled bag vector. It uses the same encoder, the same bags, and the same hyperparameter search budget, measured in trials and GPU hours. It is the most direct test of the compositional assumption.

We also re-run MILRDA on NWPU-RESISC45. That paper reports 91.56% overall accuracy, which checks that our pipeline is not weaker on standard scene classification. That benchmark is not where we expect a gain; matching it is sufficient.

On UBC, the reference is the paper's supervised Beijing functional AP of 14.0 from Urban Building Classification (UBC). Our bag-supervised model will be compared at building level using the patch-to-building assignment. Approaching that number with block labels only would be a meaningful result. Exceeding it is not expected.

Two ablations isolate the key ingredients, with everything else fixed. The first swaps the importance-pooling encoder for a standard strided one. The second swaps the trees for a multilayer perceptron fed the identical count vector. The second ablation separates "counts are the right representation" from "trees are the right reader of counts."

Why not just…?

Why not just pool embeddings with mean, max and quantiles and hand them to XGBoost? This is the most serious threat. It shares our tree aggregator and drops only the part vocabulary. If it matches our model on the confusable pairs, then the compositional representation adds nothing, and we will say so. The two models will be run on identical frozen embeddings with identical tree tuning budgets. Our prediction is that count features win specifically where classes share parts but differ in proportions. Pooled embedding statistics mix part identity and part frequency into the same coordinates, which leaves trees no clean axis to threshold.

Why not just use attention MIL with metadata concatenated? It is differentiable and standard. However, the research file documents attention drifting onto co-occurring, non-diagnostic background, and MILRDA itself reports overfitting when attention depth grows. The matched comparison described above settles this question. We will also inspect whether attention weights concentrate on parking and pavement patches in mall and park bags.

Why not just use a vision-language or OpenStreetMap-grounded foundation model? GeoLink raises urban functional zone mIoU from 55.40% to 60.00% by fusing OpenStreetMap vectors with imagery, but it depends on map coverage that is missing in exactly the poorly mapped regions where satellite-only mapping matters. SocioReasoner is too slow for national-scale mapping and is biased toward single instances. We do not claim to beat them where maps are dense. We claim a fast, image-plus-metadata model that degrades gracefully where maps are absent. We will report per-image latency alongside accuracy.

Why not just raise input resolution or add a feature pyramid? A Multi-Branch Feature Fusion Network for Building Detection in Remote Sensing Images shows standard pyramid detectors still lose small buildings when downsampled, and high-resolution crops are costly. Our encoder ablation measures directly whether importance pooling is needed at fixed patch size, or whether a plain encoder at higher resolution recovers the same part vocabulary at higher compute.

Proposed Experiments

The primary experiment is fMoW average F1, plus macro F1 restricted to confusable class pairs, comparing all baselines under a matched tuning budget.

The decisive resolution check uses fMoW's natural variation in ground sample distance, and synthetic downsampling of the finest images to 2× and 4× coarser resolution. The compositional prediction is falsifiable. The advantage of counts read by trees over attention MIL should shrink as resolution coarsens. It should also shrink when the importance-pooling encoder is replaced by a strided one. In addition, the part types that carry the most tree split gain should be those whose patches contain the smallest structures. If the advantage is flat across resolution, the gain comes from aggregation alone and the resolution story is wrong.

The localization check on UBC asks whether block-level training yields building-level functional predictions and how close they come to supervised AP 14.0. The efficiency check reports latency and parameters against MILRDA's 0.66M parameters and SocioReasoner's 2.71 s per sample.

The final analysis asks whether the trees' top split paths on confusable pairs can be read as rules about part counts. These rules would be checked by visualizing their member patches. If they cannot be read this way, the claim of interpretable composition fails, even if accuracy improves.