Ask a vision-language model "what species is this?" and then "is this a plant or an animal?". It may name the species correctly and still get the kingdom wrong. Reword either question, or shuffle the answer options, and the answers change again. Two literatures document these failures separately. One studies hierarchical inconsistency: a model is right at the fine-grained level of a taxonomy and wrong at a coarser level above it. The other studies prompt sensitivity: answers change under rewordings that should not matter. This note proposes treating both as failures of a single kind of statement, conditional independence. A conditional independence statement says that once one variable is fixed, a second variable carries no information about a third. Taken this way, the old question of which independence statements logically follow from which others becomes a practical tool. It can split a model's brittleness at coarse levels into two parts that can be measured and acted on: brittleness inherited from its fine-grained answer, and brittleness the prompt injects directly into the coarse answer.
The two-sentence version is as follows. If a model's coarse answer depended on the prompt only through its fine-grained answer, then robustness at the leaf would guarantee robustness at every ancestor. Measuring how far this "routing" condition fails gives a per-model, per-level diagnosis of where prompt brittleness enters the taxonomy. That diagnosis tells us whether to fix the model's knowledge, its prompts, or the way its coarse answers are derived.
Core Claim
Let $X$ be the input content (an image plus the underlying question intent). Let $P$ be the prompt variant, drawn uniformly by us from a designed set of paraphrases, option orders and label symbols. Let $Y_\ell$ be the model's answer at taxonomy level $\ell$, and let $Y_{\text{leaf}}$ be its species-level answer. Prompt robustness at level $\ell$ is the statement $Y_\ell \perp P \mid X$, a formalization the research file notes is implicit across ProSA, POSIX and Chen et al.. None of these papers asks what such statements imply for one another.
The semi-graphoid rules give the implication we need. From leaf robustness, $Y_{\text{leaf}} \perp P \mid X$, and routing, $Y_\ell \perp P \mid X, Y_{\text{leaf}}$, contraction yields $(Y_\ell, Y_{\text{leaf}}) \perp P \mid X$. Decomposition then yields ancestor robustness. The quantitative counterpart comes from the chain rule for conditional mutual information:
$$I(Y_\ell; P \mid X) \;\le\; \underbrace{I(Y_{\text{leaf}}; P \mid X)}_{\text{inherited}} \;+\; \underbrace{I(Y_\ell; P \mid X, Y_{\text{leaf}})}_{\text{routing failure}}$$
Here $I(A;B\mid C)$ is how many bits $B$ tells us about $A$ once $C$ is known. The first term on the right is brittleness inherited from the leaf answer. The second is brittleness that bypasses it. For a hierarchically consistent model, $Y_\ell$ is a function of $Y_{\text{leaf}}$ and the routing term is exactly zero.
The substantive claim is empirical and falsifiable. Tan et al. show that vision encoders preserve taxonomic structure and that the language model is the bottleneck. If that is right, the routing term should dominate at coarse levels in modern vision-language models. This would be a sharper statement than "the model lacks taxonomic knowledge": the prompt reaches the coarse answer through a path that does not pass through what the model believes the object is.
The inequality holds as a mathematical fact and is not itself the contribution. The contribution is the audit it makes possible, together with the implications it rules out. Li proved general conditional independence implication undecidable, so we do not aim for a complete calculus. We work only with Shannon-type derivations, which are sound but incomplete, over a small, tree-structured family of variables. To show that a claim does not follow from the measured statements, we use the falsification test of Niepert et al.: a candidate implication is disproved when the target statement's semi-lattice is not covered by the union of the premises' semi-lattices.
One example addresses the open question in the file's section 2.3: does robustness to paraphrase imply robustness to option reordering? For independent prompt factors $P_{\text{para}}, P_{\text{order}}$, the statement $Y \perp P_{\text{para}} \mid X$ does not imply $Y \perp P_{\text{order}} \mid X$. The lattice test certifies this non-implication. What does follow is that $Y \perp P_{\text{para}} \mid X, P_{\text{order}}$ together with $Y \perp P_{\text{order}} \mid X$ implies joint robustness. This tells an auditor exactly which conditional tests must be run, rather than which marginal tests happen to be convenient.
Research Program
The first component is a structured prompt design. Each query is generated from a product of factors: instruction paraphrase (several intent-preserving templates), option order, option label symbols (letters versus rare symbols, following When Benchmarks are Targets), and politeness or trailing-token noise (following The Butterfly Effect of Altering Prompts). Because we randomize $P$ ourselves and independently of $X$, the independence statements have a clean causal reading. This is also why the setup avoids the faithfulness problems raised by Drton et al. for statements inferred from observational data.
The second component is the taxonomy. We reuse the taxonomic question sets of Tan et al. on iNaturalist-2021 (kingdom to species), CUB-200-2011 and Food-101. Every level is queried under every prompt cell.
The third component is estimation. Answers are discrete multiple-choice labels, so the conditional mutual information terms are computed from per-item contingency tables and averaged over items. We apply a bias correction, since plug-in estimates are biased upward at small sample sizes. Alongside the information terms we report their flip-rate analogues: the fraction of prompt pairs that change the answer at level $\ell$, and the fraction that change it while the leaf answer stays fixed. Both are easy to interpret and do not depend on estimator choices.
The fourth component is an implication-driven audit. We enumerate the independence claims over levels and prompt factors. We derive which claims follow from which others under the semi-graphoid rules plus the functional dependencies induced by the taxonomy. We then compute a minimal set of claims to test directly, and check every derived claim against brute-force measurement, so that derivation errors or estimator bias show up.
The fifth component uses the diagnosis to choose an intervention. Where the inherited term dominates, the leaf is the problem, and only better leaf knowledge or leaf-level robustness helps. Where the routing term dominates, forcing coarse answers to be derived from the leaf answer should help. We test this prediction model by model.
Baselines
The naive reference point is the standard practice: one fixed template per level, reporting leaf accuracy and Hierarchical Consistent Accuracy (HCA), the fraction of items answered correctly at every level of the path. Tan et al. report 55% leaf accuracy and 32.8% HCA for Qwen2.5-VL-72B on iNat-Plant, with more than 67% of hierarchical paths failed. We rerun this exact protocol on the same models so that our prompt-randomized numbers sit next to a reproduced fixed-prompt anchor.
The strongest measurement competitor is brute-force per-level sensitivity scoring. This means running a sensitivity index such as POSIX, or the consistency metrics of CARETS, at every level on the full prompt grid. CARETS reports 10–15% drops under option swapping and negation consistency of 18–28%. We found no result reported in the research file for POSIX on hierarchical question sets. This baseline uses the same forward passes as our method, so compute is matched exactly. Our method beats it in two cases. The first is if the decomposition predicts which intervention reduces coarse-level brittleness when total sensitivity scores cannot. The second is if the reduced test set recovers the full-grid sensitivity profile within estimator error at a fraction of the calls.
The strongest competitor for predicting individual flips is confidence-based. ProSA ties brittleness to flat output distributions, and Chen et al. report prompt sensitivity correlating with correctness at Pearson $r$ up to −0.40. We give both methods the same held-out prompt cells. Our method beats them if the routing term adds predictive power for ancestor flips (measured by area under the ROC curve) beyond ancestor and leaf confidence combined.
The strongest method-level fix is ProTeCt. It raises HCA on ImageNet from 2.99% (CoOp) to 37.74%, and Mean Treecut Accuracy from 46.98% to 88.61%, by tuning soft prompts across tree levels in CLIP. It applies only to contrastive models. It therefore serves as a reference on the ImageNet taxonomy, where we also evaluate CLIP with and without ProTeCt-style tuning. We ask whether its gains appear in the inherited term, in the routing term, or in both.
The key ablation removes the conditioning on the leaf answer. We attribute ancestor brittleness using marginal sensitivities alone and check whether the intervention choice it recommends is worse than the one the decomposition recommends.
Why not just…?
Why not just measure sensitivity at every level directly? It is affordable at this scale, and we do it, as the ground truth. A per-level total cannot tell whether a coarse flip comes from an unstable leaf answer or from a prompt path that bypasses the leaf. That distinction determines the fix. The measurement baseline above tests whether the distinction pays off. If the decomposition never changes which intervention wins, we would report that the extra structure is unnecessary.
Why not just derive coarse answers from the leaf answer? This is a serious threat as a fix. Xu et al. find leaf-propagated mAP of 58.73 against 24.91 for direct prediction in CLIP on the WordNet hierarchy. Leaf-then-derive prompting forces the routing term to zero by construction. It also caps ancestor accuracy at leaf accuracy and imports all leaf brittleness. Our framework predicts that it helps exactly when the routing term exceeds the leaf-induced loss. We include leaf-then-derive prompting as an arm and test that prediction. We do not claim to beat it everywhere.
Why not just use output confidence to flag unstable answers? Confidence costs one pass and is well supported by ProSA. A confident but wrong coarse answer that tracks the prompt rather than the leaf answer is the case confidence cannot see. The flip-prediction comparison checks whether that case is frequent enough to matter.
Proposed Experiments
The first experiment measures the decomposition. On iNaturalist-2021, CUB and Food-101, across Qwen2.5-VL at several sizes and the other models in Tan et al., we estimate the inherited and routing terms at every level under the full factorial prompt design. The decisive check is whether the routing term grows toward coarse levels. If the routing term is near zero everywhere, the "LLM bottleneck" reading is wrong for prompt brittleness, and the brittleness is inherited from the leaf.
The second experiment tests the audit's implications. We certify the paraphrase/reorder non-implication with lattice exclusion and confirm empirically that models robust to paraphrase still flip under reordering. We then show that the derived minimal test set predicts the untested claims. The decisive failure would be a derived claim that brute force contradicts beyond the estimator's confidence interval.
The third experiment tests prediction-guided intervention. For each model and level, the decomposition picks among no change, leaf-then-derive prompting, and prompt ensembling. We compare HCA and the flip rates it produces against a rule that always applies the single best intervention on average.
The fourth experiment is a text-only transfer. Tan et al. report $\rho \approx 0.91$ between text-only taxonomy knowledge and multimodal consistency. We run the same decomposition on text-only noun hierarchies from AbsPyramid. We test whether each model's text-only routing term predicts its multimodal routing term, which would let us diagnose a vision-language model without images.
References
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- POSIX: A Prompt Sensitivity Index For Large Language Models
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- Vision LLMs Are Bad at Hierarchical Visual Understanding, and LLMs Are the Bottleneck
- Undecidability of Network Coding, Conditional Information Inequalities, and Conditional Independence Implication
- On the Conditional Independence Implication Problem: A Lattice-Theoretic Approach
- When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards
- The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance
- Faithlessness in Gaussian graphical models
- CARETS: A Consistency And Robustness Evaluative Test Suite for VQA
- ProTeCt: Prompt Tuning for Taxonomic Open Set Classification
- Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity
- AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graph