When the Camera Misses the Attribute: Evidence-Aware Feasibility for Open-World Compositional Recognition

Compositional recognition asks a model to name things like "peeled apple" or "cracked mug" by combining a state (or attribute) with an object, including pairs never seen in training. In the "open-world" version of the task, the model must choose among every state–object pair it knows. That is hundreds of thousands of candidates, most of them absurd, like "ripe dog". The main fix has been a feasibility score: a number saying how plausible each pair is, used to suppress the absurd ones. Every such method assumes one complete image. Real systems often see an object through several cameras or frames, and some of those views drop out. The key asymmetry is that objects are usually recognizable from almost any view, while states often are not. You cannot tell that an apple is peeled, or a shoe is suede-toed, from the side that hides it.

We propose to make the feasibility prior do more work exactly when the observed views cannot reveal the state, and less when they can. Concretely, the model estimates how likely each state is to be visible from each observed view. This estimate, which we call evidence coverage, decides how much the final score trusts visual state evidence and how much it trusts the plausibility prior. A missing view then leads the model to predict "the most plausible state for this object". The alternative is to let a weak, confounded cue in the remaining view pick an implausible pair, which is the failure that CAILA documents for CLIP-style matching. The note also builds the first benchmark where compositional zero-shot recognition and missing views meet, because none exists today.

Core Claim

Existing open-world methods apply one fixed feasibility weighting, tuned for complete single images. Under missing views, this weighting is miscalibrated in a predictable direction. Too little prior is used when state evidence is absent, and it is used indiscriminately when state evidence is present. We claim that weighting the prior per candidate pair, by predicted evidence coverage, improves open-world accuracy under view dropout without hurting the full-view case. We further claim that the gain grows as fewer views are observed. We would treat a flat or shrinking gain as a refutation.

The scoring rule is simple. Let $X_{obs}$ be the set of observed views and $x_v$ a single view. The score of state $s$ and object $o$ is

$$\text{score}(s,o) = \log p(o \mid X_{obs}) + \alpha(s,o)\,\ell(s \mid o, X_{obs}) + \big(1-\alpha(s,o)\big)\log g(s,o),$$

The terms are as follows. $p(o \mid X_{obs})$ is an object posterior pooled across whichever views are present. $\ell(s \mid o, X_{obs})$ is an object-conditioned state log-likelihood from the observed views. $g(s,o)$ is a feasibility score. The coverage weight is

$$\alpha(s,o) = 1 - \prod_{v \in X_{obs}} \big(1-\rho_v(s,o)\big),$$

where $\rho_v(s,o)$ is the predicted probability that state $s$ on object $o$ would be visually detectable in view $v$. The product form is the standard noisy-OR: the state goes unseen only if every observed view fails to show it. It makes adding a view never reduce coverage.

Relation to Prior Work

The feasibility side supplies our method. Open World Compositional Zero-Shot Learning (CompCos) derives feasibility from visual similarity between primitives and injects it as a training margin. Its authors showed this beats test-time masking, and we keep the margin during training. Learning Graph Embeddings for Open World Compositional Zero-Shot Learning (Co-CGE) propagates feasibility through a graph over states and objects. Feasibility with Language Models for Open-World Compositional Zero-Shot Learning (FLM) reads feasibility off a language model's "Yes" logit and beats GloVe and ConceptNet priors. TsCA uses transport costs between patches, compositions and primitives as a feasibility filter. All four apply feasibility identically regardless of how much visual evidence an input carries. That is the single thing we change, and we reuse FLM-style scores as $g$.

The state side also supplies method and constrains our design. Attributes as Operators and CANet model states as object-dependent transformations or object-conditioned embeddings. That evidence rules out a pure product of independent state and object classifiers, so our $\ell$ conditions on $o$. Seeing the Unseen: Composing Outliers (COMO) routes between seen and unseen heads by confidence. This motivates a specific risk: under missing views, low confidence from absent evidence can be mistaken for low confidence from a novel pair. Our coverage term separates the two.

The multi-view side motivates the work. DMCAR separates shared "public" experts from per-view "private" experts. It names incomplete views as unsolved, because missing views silence private experts and distort its pooled routing context. CL2P learns consistent-then-specific features and likewise defers incomplete data. Our object posterior plays the role of a public consensus. We deliberately avoid per-view private expert pools, which DMCAR notes scale poorly. Instead we use one shared state head plus a small visibility predictor $\rho_v$. We also borrow the lesson of DealMVC: align samples through confident class-level targets rather than strict view-to-view pairing when view sets differ. Both multi-view papers treat classes as monolithic cluster indices, and none tests unseen combinations.

Research Program

The model has three parts. The first is a shared image encoder applied per view, with a CLIP-style or single-stream backbone. The single-stream design of An Integrated Approach to Open-World CZSL is a candidate; its sparse compositor keeps the pair head at $2|A||O|$ parameters. The second part is the object posterior, obtained by attention pooling over present views only, with no imputation. The third is an object-conditioned state head together with a visibility head producing $\rho_v(s,o)$ from view features and the object embedding.

Visibility is supervised without extra labels on complete multi-view training data. For each view, a single-view state classifier is scored against the ground-truth state. The view is marked as revealing the state if it ranks the true state within its top few among states feasible for that object. $\rho_v$ is trained to predict this indicator. Training applies random view dropout to all components, and to every baseline, so robustness to dropout is not credited to our scoring rule.

A benchmark is required, and building it is half the contribution. The primary testbed uses OmniObject3D, which provides 6,000 real scans across 190 categories and 100 rendered views per object (OmniObject3D). We would annotate each scan with state and attribute labels of the kind PACO uses: color, material, pattern and reflectance. Where categories allow, we would add physical states such as open, closed, broken or peeled. We would then withhold pairs to form a compositional split and evaluate over the full state-by-object product.

Views are sampled from fixed azimuth bins, so a "missing view" is a real missing camera. This gives a natural gradient: some attributes are view-invariant (overall color) and others one-sided (a printed pattern). The annotation cost is real. The 3D pipeline in OmniObject3D's own sparse-view study shows three views often under-constrain geometry, which is the regime we care about.

As a secondary stress test, we convert MIT-States and C-GQA into a multi-region form in which image quadrants act as views and are dropped. This is artificial and will be reported as such. Its main use is to confirm that we match baselines when all regions are present.

Baselines

The naive reference is off-the-shelf CLIP scoring every pair by cosine similarity. It uses the mean of available view embeddings and no feasibility prior. It shows how bad the raw open-world problem is under dropout.

The strongest practical competitor is the same late-fusion model with three additions: fine-tuning with view dropout, an FLM feasibility score as a fixed-weight test-time filter, and the CompCos feasibility margin in training. It combines the best published feasibility signal (FLM), the best published way to use it (CompCos), and the obvious robustness trick. Its fixed prior weight is tuned on a validation set that includes dropout. It gets the same backbone, data, epochs and hyperparameter search budget as our model.

We also run COMO and the single-stream sparse compositor, each fed mean-pooled views, along with a DMCAR-style public/private encoder with zero-imputed missing views. For reference, the file gives these single-image open-world numbers. CompCos reaches open-world AUC 1.6 on MIT-States. The single-stream model reaches open-world AUC 3.1 and harmonic mean 12.4 on MIT-States. Co-CGE reaches open-world AUC 0.78% on C-GQA. TsCA reaches peak open-world AUC 37.1% on UT-Zappos. COMO reports AUC 22.4 on MIT-States, though the summary does not say whether that figure is open-world, so we will not treat it as comparable until verified. On the new multi-view benchmark there is no reported result found on Lacuna for any method.

Beating the strongest baseline means two things. At one and two observed views, our open-world AUC and harmonic mean must be higher over three seeds, with non-overlapping confidence intervals. At all views, the loss must be under half a point.

The decisive ablation replaces $\alpha(s,o)$ with a single scalar tuned per number of observed views. That variant knows how many views are missing but not which states they hide. If it matches the full model, pair-specific coverage is not the source of the gain, and the claim reduces to "tune the prior weight by view count".

Why not just…?

Why not just train a standard model with view dropout and a fixed feasibility filter? This is the real threat. It is plausible that dropout training teaches the encoder to hedge, so that a fixed prior suffices. The per-view-count scalar ablation above settles it. If the scalar variant recovers most of the gain, we will report that the simpler recipe is sufficient. We expect coverage to matter mainly for one-sided attributes. We therefore report results split by an attribute's measured view-dependence, where a real effect should concentrate.

Why not just multiply independent state and object classifiers and apply them to whatever view remains? State appearance depends on the object, as CANet and Attributes as Operators argue. The integrated single-stream work likewise reports that independent recognition misses compositional context. More importantly, the independent product has no mechanism that tells "state not visible" apart from "state absent". We run it as a baseline.

Why not just mask impossible pairs at test time with ConceptNet or GloVe? CompCos found training-time margins beat test-time masking, FLM found language-model feasibility beats both static sources, and the LE+ baseline collapses to 0.08% open-world AUC on C-GQA (Co-CGE). A static mask also cannot vary with the evidence, which is the quantity we argue should drive the prior's weight.

Why not just impute the missing views with a reconstruction model? Hallucinated views invent state evidence. The imputed state then comes from appearance statistics, not from the object's plausible states. We will test this directly with a feature-imputation baseline. We will also measure its "distractor rate": the fraction of top-1 predictions whose FLM feasibility falls below the median.

Proposed Experiments

The central experiment plots open-world AUC, harmonic mean, and seen and unseen accuracy against the number of observed views, from one up to all, on the OmniObject3D split. It compares all baselines on identical view masks. The prediction is a gap that widens as views drop. A constant gap would mean coverage adds nothing beyond a better backbone.

A second check tests whether $\rho_v$ means what we say. On held-out objects, predicted coverage should correlate with the single-view state classifier's actual accuracy per view. Performance should improve most on pairs whose true state is one-sided.

A third check targets the confusion flagged by COMO: under dropout, how often does a seen pair get routed to the unseen head? Our model should reduce this relative to COMO fed mean-pooled views.

Finally, on the region-dropout versions of MIT-States and C-GQA, the full-input results must stay within noise of our reproduced baselines. Otherwise the method is buying multi-view robustness at the cost of the standard task.