Overcoming the Granularity Gap in Vision Language Models

65 concepts58 papers3 proposals

State-of-the-art vision-language models (VLMs) demonstrate an extraordinary capacity for zero-shot generalization across broad semantic categories. Show a foundational VLM an image of a dog versus a car, and it distinguishes them effortlessly. However, ask that same model to differentiate between two highly similar sub-categories—such as specific plant diseases, subtle fabric weaves, or precise vehicle sub-models—and its performance hits a definitive ceiling.

This phenomenon is known as the domain-specific granularity gap. As Jingjing Chen et al. demonstrate in Efficient Prompt Tuning for Hierarchical Ingredient Recognition, while foundational models excel at identifying coarse parent categories in a zero-shot setting, they lack the specialized vocabulary and visual sensitivity required for precise sub-category identification.

To build systems capable of expert-level discrimination, researchers are uncovering why general-purpose models fail at the tail end of the data distribution and developing mechanisms to bridge this granularity gap.

The Geometry of Failure: Blind Pairs and Coarse Optimization

Why do general-purpose VLMs fail when transitioning from broad concepts to domain-specific tasks? The root cause lies in the geometry of the shared embedding space learned during web-scale contrastive pretraining.

Contrastive learning forces models to map images and text captions into a joint space by maximizing the cosine similarity of matching pairs and minimizing it for non-matching pairs. Because web data predominantly features "storytelling" or conversational captions rather than technical, attribute-heavy descriptions, the model optimizes for global semantic context.

This optimization strategy creates what Jihai Zhang terms blind pairs in CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling—semantically distinct images that standard encoders map to nearly identical representations. As Pan Zhang et al. note in RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognition, foundational models lack the specialized discriminative power required to resolve these fine-grained visual conflicts because their embedding spaces collapse subtle intra-class variations.

Coarse Pretraining Space:
[Image: Crewneck] ─── (High Similarity) ─── [Image: V-neck]
       │                                           │
       └─────────── Both map near "Sweater" ───────┘

Fine-Grained Aligned Space:
[Image: Crewneck] ─── (Low Similarity) ──── [Image: V-neck]
       │                                           │
  (Aligned)                                   (Aligned)
       ▼                                           ▼
"Crewneck Sweater"                         "V-neck Sweater"

This structural limitation manifests in two distinct ways across specialized domains:

  1. Visual Micro-feature Blindness: In fields like forensics or manufacturing, the critical differentiator is often a minute texture or pattern. Mihir Chauhan et al. demonstrate in Vision-Language Model Based Handwriting Verification that general VLMs struggle to capture the subtle micro-features required for expert-level verification. Similarly, Suvir Mirchandani et al. observe in FaD-VLP: Fashion Vision-and-Language Pre-training towards Unified Retrieval and Captioning that distinguishing highly similar attributes like necklines or sleeve lengths requires a level of detail that broad pretraining completely overlooks.
  2. Linguistic Mismatch: General-purpose knowledge bases lack the highly specific, technical descriptions required to separate technical sub-categories. As Han Xiao et al. explain in FashionViL: Fashion-Focused Vision-and-Language Representation Learning, specialized domains feature a high density of fine-grained attributes and nouns rather than narrative structures. When models encounter raw web documents, the inherent linguistic noise drowns out these critical technical terms, a challenge addressed by Lingqiao Liu et al. in Less is More: Zero-Shot Learning from Online Textual Documents with Noise Suppression.

Injecting Discriminative and Structured Semantics

To force the embedding space to separate closely related concepts, researchers have moved away from generic template prompts (such as "a photo of a [class]") toward structured, attribute-rich semantic guidance.

The core intuition is that if the model is given explicit, discriminative features to look for, it can leverage its latent visual capacities more effectively. Muzammal Naseer et al. pioneer this in S3A: Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment by using Large Language Models (LLMs) to generate category-specific discriminative prompts, providing the precise semantic guidance needed to resolve ambiguity in massive label spaces.

Instead of treating labels as isolated text strings, modern frameworks structure domain knowledge into dense attribute matrices or tables. For example, in industrial textile retrieval, Francesco Giuliari et al. show in An analysis of vision-language models for fabric retrieval that prompting Multimodal LLMs to generate structured, attribute-rich technical descriptions significantly boosts retrieval accuracy over natural language descriptions.

Taking this structural alignment further, Ada-Astrid Balauca et al. introduce Taming CLIP for Fine-Grained and Structured Visual Understanding of Museum Exhibits, which maps generic visual embeddings directly into structured tabular data. This allows a model to simultaneously extract an artifact's material, era, and production technique, forcing a multi-dimensional alignment that flat text strings cannot achieve.

Forcing Localized Visual Attention

Altering the text side of the equation is only half the battle; the visual encoder must also be re-engineered to prioritize local micro-features over global silhouettes.

Standard Vision Transformers (ViTs) tend to focus on dominant spatial regions and global shapes. To counteract this, researchers introduce localized masking and multi-scale visual processing. In Masked Vision-language Transformer in Fashion, Mingchen Zhuge et al. introduce a Masked Image Reconstruction (MIR) task that forces the model to reconstruct missing pixel-level details. By stripping away global context, the model is forced to learn fine-grained fabric textures and design features at the pixel level. The authors expand on this multi-scale intuition in Kaleido-BERT: Vision-Language Pre-Training on Fashion Domain, utilizing a "kaleidoscope" of multi-scale image patches combined with alignment-guided masking to stitch local details directly to specific technical descriptors.

Standard ViT Processing:
[ Full Image ] ───► Global Self-Attention ───► Focuses on overall silhouette

Masked / Multi-Scale Processing (e.g., Kaleido-BERT):
[ Patch: Collar ] ───┐
[ Patch: Texture] ───┼─► Localized Masking ───► Forces pixel-level & attribute alignment
[ Patch: Sleeve  ] ───┘

Finally, the contrastive learning objective itself must be modified to penalize fine-grained confusion. Standard contrastive loss treats all negative examples equally—an image of a truck is just as much of a negative to a "sedan" query as an image of a slightly different sedan sub-model.

To fix this, Wonyoung Shin et al. develop e-CLIP: Large-Scale Vision-Language Representation Learning in E-commerce. They introduce category-based negative sampling, which deliberately selects hard negatives from within the exact same product category. This forces the encoder to actively carve out distinct sub-spaces for items that are visually and textually similar but functionally distinct, effectively closing the granularity gap.

Key Foundational Work

Deep domain adaptation for describing people based on fine-grained clothing attributes (2015) — contributed 1 concept elements to this direction.
VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback (2016) — contributed 1 concept elements to this direction.
Business-Aware Visual Concept Discovery from Social Media for Multimodal Business Venue Recognition (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_domain_specific_granularity_gap_2f2ac89a,
  title = {domain-specific granularity gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/domain-specific-granularity-gap/rdr_2f2ac89a17394cc5af34a01755bef0f8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_domain_specific_granularity_gap_2f2ac89a_v1,
  title = {domain-specific granularity gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/domain-specific-granularity-gap/rdr_2f2ac89a17394cc5af34a01755bef0f8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake