Image-text models such as CLIP separate "dog" from "car" easily but confuse closely related species, aircraft variants, or diseases. Recent fixes embed the class taxonomy as a partial order: each concept becomes a cone, and a child's cone sits inside its parent's. These methods only constrain a concept from above. Its ancestors bound it, but nothing pins it down from below. In order theory, an element is reliably reached as a limit only when it can be approximated from both sides at once, by an increasing family below it and a decreasing family above it. When only one side exists, "converging in the order" and "being nearby in distance" come apart. We propose that the same mismatch explains a documented failure in fine-grained vision-language learning: hyperbolic cone objectives keep the hierarchy but collapse siblings together. Current work patches this by adding a second geometry.
We propose a single training objective in one hyperbolic space. It makes every concept both the greatest lower bound of its ancestors, meaning the image is excluded from every non-ancestor cone, and the least upper bound of its own image instances, meaning the concept's cone fits its members tightly. We pair this with a diagnostic, the order–metric discrepancy. It is the fraction of images whose deepest containing cone disagrees with their nearest class embedding. In two sentences: fine-grained classes collapse because the embedding pins each class down only from the coarse side. Adding a "tightness from below" term should restore sibling separation without a second geometry and make order-based and distance-based predictions agree.
Where the two directions actually meet
The order-theory direction says when order convergence coincides with convergence in some topology. O1-convergence in partially ordered sets proves that $O_1$-convergence is topological exactly when the poset is B-consistent and $S^*$-doubly continuous. The underlying lesson is that elements must be approachable both from below, by directed sets, and from above, by filtered sets. The Equivalence of Two Modes of Order Convergence settles Wolk's problem with "ID-pairs": a lower set and an upper set that meet exactly at one point. It shows that $o$- and $o_2$-convergence agree exactly on ID-doubly continuous posets.
We take only the lesson from these papers, not their theorems. The same paper also shows that every finite poset is ID-doubly continuous. A finite taxonomy is therefore trivially well-behaved, and "enforcing ID-double continuity on the class tree" would be vacuous. That rules out the obvious design. The non-trivial object is the continuous cone order on the embedding space, restricted to the points the model actually produces. That order can fail two-sided approximation, and we can measure whether it does.
The vision-language direction supplies the failure and the methods. Hierarchical Image Classification using Entailment Cone Embeddings introduced Poincaré cones for class hierarchies with a CNN backbone. Hyper3-CLIP applies Lorentz-model entailment to CLIP training with query-conditioned pooling. HiR² regularizes multimodal LLM layers with hyperbolic entailment and found that entailment alone collapses siblings, so it adds a spherical dispersive loss. All three constrain concepts only from above via containment. HiR²'s finding is the direct motivation for this proposal.
Hierarchy-Aware Fine-Tuning of Vision-Language Models takes a probabilistic route. It enforces path and sibling consistency with TP-KL and HiSCE losses over a fixed taxonomy, without any embedding order. HiMo-CLIP enforces only a scalar monotonicity, where richer text should score higher. We differ from all of these by adding the missing lower-side constraint, testing whether it removes the need for a second geometry, and making the order/metric agreement a measured quantity rather than an assumption.
Core Claim
Let each concept $c$, whether a leaf or an internal node, have a text-derived point $t_c$ in the Lorentz model. Let $\mathrm{ext}(t_c, x)$ be the exterior angle at $t_c$ toward an image embedding $x$, and let $\psi(t_c)$ be the cone half-aperture at $t_c$. The aperture narrows as $t_c$ moves away from the origin. The image $x$ lies in $c$'s cone iff $\mathrm{ext}(t_c,x)\le\psi(t_c)$.
The objective has three terms.
The upper-bound term is the standard containment penalty $\max(0,\mathrm{ext}(t_a,x)-\psi(t_a))$. It applies to every ancestor $a$ of the image's label, and to the label itself.
The exact-meet term makes the set of cones containing $x$ meet exactly at $c$, which is the ID-pair condition read on the model's outputs. It is the penalty $\max(0,\psi(t_s)-\mathrm{ext}(t_s,x)+m)$ for every non-ancestor concept $s$, with sibling cones drawn as hard negatives and margin $m$.
The new lower-bound or tightness term asks $c$ to be the least cone containing its members $X_c$ in the batch:
$$ \mathcal{L}_{\text{tight}}(c)=\psi(t_c)-\max_{x\in X_c}\mathrm{ext}(t_c,x), $$
This quantity is non-negative whenever containment holds, and it measures unused aperture, the slack. Internal nodes apply the same term using their children's points as members. A parent therefore becomes the join of its children, not just some cone containing them.
The hypothesis follows. Tightness pulls each concept toward its own evidence. Siblings with different evidence should then occupy narrow, non-overlapping cones in the same space, which removes the reason HiR² needed a sphere.
Prediction and the diagnostic
At test time we form two predictions. The order limit is the deepest concept whose cone contains $x$, meaning the infimum of $x$'s upper bounds, with a backoff to the nearest boundary when no leaf contains it. The metric limit is the nearest leaf by Lorentz distance.
Their disagreement rate $D$ is our measurable version of order-topological discrepancy. We predict that $D$ tracks the hierarchical inconsistency metrics used in prior work, and that tightness lowers $D$ more than the other terms do. Concepts are text embeddings, so a held-out species can be inserted as a new cone under a known parent at test time. This open-taxonomy case is where closed-set methods have no answer.
Research Program
We first measure $D$ across existing models, both zero-shot CLIP and entailment-only fine-tuned CLIP, on CUB-200, FGVC-Aircraft and iNat21. We test whether $D$ correlates with the Tree-based Inconsistency Error (TICE) and Full-Path Accuracy (FPA) per class. A null correlation would weaken the motivating story before any method is built.
We then fine-tune a CLIP ViT-B/16 dual encoder in Lorentz space with LoRA under the three-term objective. Separately, we port the objective into HiR²'s setting, a Qwen2.5-VL backbone with supervised fine-tuning on iNat-Animal. There we replace HiR²'s two-geometry regularizer with our one-geometry version. This second setting is where the core claim is decided. A minor extension, outside the main claim, tests whether the lower bound helps Hyper3-CLIP's prompt sensitivity. In that extension, the lower bound is supplied at inference by crops of the image placed under the image's own point, rather than by a discarded pooling module.
Baselines
The naive reference points come first. One is zero-shot CLIP with class-name prompts, and the other is flat LoRA cross-entropy on leaf classes. The flat baseline is exactly what a skeptic would scale up. Hierarchy-Aware Fine-Tuning of Vision-Language Models reports its flat baseline at 50.2% FPA on CUB-200. We re-run both on our backbone rather than borrow that number, because the file does not state the backbone behind it.
The strongest dual-encoder competitor is the TP-KL plus HiSCE method from that same paper. It reports FPA rising to 71.1% on CUB-200 and TICE falling from 17.9 to 8.5 on FGVC-Aircraft. Beating it means higher leaf accuracy and FPA, with TICE no worse, on matched LoRA rank, epochs, data and an equal hyperparameter search budget of the same number of trials per method. On held-out leaves, this method cannot score new classes at all, so the comparison there is against zero-shot CLIP and the entailment-only model.
The strongest competitor in the multimodal-LLM setting is HiR² ( Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models). It reports Hierarchical Consistent Accuracy (HCA) rising from 14.06 to 18.07 on iNat-Animal with Qwen2.5-VL under supervised fine-tuning. We reproduce it with its two geometries and run ours with the same layers, schedule and tuning budget. The claim succeeds if the one-geometry objective matches or exceeds HiR²'s HCA and leaf accuracy, within seed variance over three seeds. Matching HiR² while dropping the sphere is the substantive outcome; exceeding it is a bonus.
The ablations isolate the key ingredient. Entailment-only, using the upper-bound term alone, reproduces the sibling collapse HiR² describes. Upper plus exact-meet without tightness is the obvious "just add negatives" model. The full objective is compared against both. A marginalization classifier, which predicts the leaf and then reads off its ancestors, is included as a consistency ceiling. Entailment cones reports such a classifier at 0.9223 mean F1 on ETHEC, against 0.5977 species-level F1 for their hyperbolic cones.
Why not just…?
Why not just add sibling negatives to entailment cones? This is the real threat. The exact-meet term already pushes images out of sibling cones, and that may be all that matters. The ablation of upper plus exact-meet versus the full objective settles it. If tightness adds nothing to leaf accuracy, HCA or $D$, then the lower-bound idea fails. We would report that the order-theory lens reduced to standard hard-negative mining.
Why not just keep HiR²'s second spherical geometry? It works, per HiR², but it splits one concept into two coordinates. The order a model retrieves with then differs from the space it separates siblings in, which is exactly the kind of order/metric split that $D$ measures. If our single space matches HiR²'s HCA, it is the simpler model. If it does not, the result is still informative, because it would show that the second geometry does work tightness cannot.
Why not just predict the leaf with flat fine-tuning and read off the ancestors? In a closed taxonomy, this gives perfect consistency by construction. Consistency metrics alone therefore cannot justify our method. That is why our headline numbers are leaf accuracy and held-out-leaf zero-shot accuracy. The readout trick cannot score a species absent from training, and flat cross-entropy's 50.2% FPA on CUB from Hierarchy-Aware Fine-Tuning shows it also loses on the full path.
Why not just expand prompts with an LLM and cluster, as S3A does? S3A lifts StanfordDogs top-1 from 35.29% to 58.94% by clustering images and generating discriminative attributes. That approach is complementary rather than competing. LLM descriptions can serve as the concept points $t_c$ in our method, and we test this combination on StanfordDogs.
Decisive checks
Three results decide the proposal.
The first is the iNat-Animal HCA comparison against HiR²'s reported 18.07, with the full objective in one Lorentz space versus our reproduction of HiR². The second is the tightness ablation on CUB-200, FGVC-Aircraft and iNat21: if removing tightness costs nothing, the core claim is false. The third is the diagnostic test. If $D$ falls under tightness and predicts per-class TICE with clear rank correlation, the order-topological reading is doing explanatory work. If $D$ is uncorrelated with errors, the reading is decorative even if accuracy improves.
Held-out-leaf evaluation, with 20% of species removed from training on CUB-200 and iNat21 while their parents are kept, tests the open-taxonomy advantage over the probabilistic and marginalization baselines. ETHEC's four-level butterfly hierarchy provides a direct comparison point to the original cone work.
References
- O1-convergence in partially ordered sets
- The Equivalence of Two Modes of Order Convergence
- Hierarchical Image Classification using Entailment Cone Embeddings
- Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training
- Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models
- Hierarchy-Aware Fine-Tuning of Vision-Language Models
- HiMo-CLIP: Modeling Semantic Hierarchy and Monotonicity in Vision-Language Alignment
- Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models
- S3A: Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment