CAPTOR: text-encoder-free grounding by counting caption co-occurrence over the visual words of a frozen encoder

Elad Yoshai, Natan T. ShakedarXiv 2026 · arXiv

TL;DR

CAPTOR provides a high-performance text interface for "language-blind" frozen image encoders without requiring a text encoder or gradient-based training. By simply counting how terms co-occur with "visual words" in a captioned dataset, it achieves state-of-the-art training-free grounding on natural images and significantly outperforms specialized pathology models that were actually trained with language.

What Problem Does This Paper Address?

Modern vision-language models like CLIP rely on a text encoder perfectly aligned with an image encoder. However, in many specialized fields—most notably computational pathology—the best image encoders are often trained on massive unlabeled datasets (e.g., millions of tissue slides) and lack a corresponding text encoder.

To "talk" to these models, researchers usually have to train a new alignment, which is computationally expensive, requires large paired datasets, and must be repeated for every new encoder. The authors of CAPTOR ask a provocative question: Can we ground text in images using only a frozen image encoder and a pile of captioned images, without ever using a text encoder or backpropagation?

Methods

CAPTOR (Caption-Anchored Patch-Term Occurrence Representation) operates in two distinct stages:

  1. Offline Preparation:
    • Visual Words: The system takes a frozen image encoder and runs k-means clustering on patch tokens from a caption corpus to create $K$ "visual words."
    • Term Signatures: For every term in the vocabulary, it calculates a signature—essentially a standardized measure of how much more (or less) a specific visual word appears in images where the term is mentioned in the caption versus images where it isn't.
  2. Frequency Correction: To ensure rare terms don't dominate the scores, CAPTOR uses a "permutation null" (shuffling captions) to put all term signatures on a common, comparable scale.
  3. Online Inference:
    • When a new image arrives, it is passed through the frozen encoder once. Each patch is assigned to its nearest visual word (forming an image factor $\Phi$).
  4. Grounding a query term is then reduced to a simple table lookup in the pre-computed signature table ($S$). The final compatibility matrix is factorized as $A = S\Phi$.
The CAPTOR pipeline: Offline preparation builds a signature table, while online inference uses a single forward pass to map patches to words and look up signatures.

Datasets

  • Natural Images: The authors used COCO (15k-30k training captions) to build the signature table and evaluated on COCO-validation, Pascal VOC 2012, and Cityscapes.
  • Computational Pathology: The table was built using PathCap (32k captioned figures). Evaluation was performed on specialized benchmarks: BCSS (breast cancer) and LUAD-HistoSeg (lung cancer).
  • Naming Benchmarks: PanNuke and NCT-CRC were used to test how well the system could name visual words across different organs.

Key Insights and Ideas

  • Counting is a Powerful Proxy for Alignment: The core "aha moment" is that co-occurrence statistics between captions and visual clusters can replace the complex latent-space alignment of neural text encoders.
  • The Importance of the "Null": Raw co-occurrence counts are biased toward rare words. By dividing signatures by a permutation null (Figure 2), CAPTOR ensures that "person" and "handbag" can be compared fairly, even though one is vastly more common than the other.
  • Factorization for Efficiency: Because the matrix factorizes, an image is encoded exactly once. Adding a new query term doesn't require a new pass through a text encoder; it's just adding a row to a small table.
  • Domain Superiority: In specialized domains like pathology, general-purpose text encoders (like CLIP's) fail because they don't know the vocabulary. CAPTOR learns the domain-specific vocabulary directly from the captions of that domain.

Key Results

  • Natural Images: On COCO, CAPTOR reached 0.565 mIoU, significantly beating training-free baselines like ProxyCLIP (0.502). It performed on par with Talk2DINO, despite Talk2DINO using a text encoder and five times more caption data.
  • Pathology Breakthrough: This is where CAPTOR shines. On breast cancer slides, CAPTOR using the UNI encoder (which has no native language support) outperformed the strongest baseline that did use a pathology-specific text encoder by 0.132 mIoU.
  • Speed and Efficiency: Building the entire signature table takes only 4.3 seconds on a GPU (or ~1 minute on a CPU). Adding a new term to the vocabulary is roughly 30 times faster than a pass through CLIP’s text encoder.
Qualitative results on Cityscapes showing CAPTOR (third column) effectively grounding various object classes compared to baselines.

Limitations and Future Work

  • Vocabulary Constraints: CAPTOR can only ground terms that appear frequently enough in the caption corpus to clear the "permutation null."
  • Bag-of-Words Limitations: The current method does not handle complex linguistic structures like negation ("no tumor"), attributes ("red car"), or spatial relations ("cat on a mat").
  • Visual Word Resolution: The grounding is limited by the quality of the initial k-means clustering. If the frozen encoder doesn't visually distinguish between two concepts, CAPTOR cannot separate them.
  • Context Absorption: The authors noted that objects often seen in specific settings (like skis on snow) can "absorb" the signatures of their surroundings, leading to some background pixels being mislabeled as the object.

Summary updated

Abstract

Grounding a text query in an image needs a matrix scoring every term against every patch. Vision-language models compute it with an aligned text encoder, which many image encoders, such as histopathology-related models trained on slides alone, lack. The Caption-Anchored Patch-Term Occurrence Representation (CAPTOR) estimates the matrix without a text encoder. It clusters a frozen image encoder's patch tokens into visual words and describes each captioned image by its share of patches in each word. A term signature is the standardized difference of those shares between images whose captions mention the term and the rest. The matrix factorizes into a term-by-word table and a query-independent image factor, so an image is encoded once and a new term adds one table row. Against every training-free baseline, CAPTOR leads on Common Objects in Context (COCO), Pascal Visual Object Classes (VOC) and Cityscapes. On COCO, scored on the cells one object owns, it reaches 0.565 mean intersection over union (mIoU) against 0.502. On all three, it is also level with a text alignment trained on the same backbone, which needs gradients, a text encoder and several times the captions. On held-out breast-cancer slides, it exceeds the strongest baseline scoring patches with a pathology text encoder by 0.160 mIoU. Read through CAPTOR, UNI, an encoder trained on slides alone, exceeds such baselines by 0.132 on a common grid. Counting can therefore give a text interface to any image encoder trained without language, wherever captioned images exist.
BibTeX
@misc{yoshai2026captor,
  author = {Elad Yoshai and Natan T. Shaked},
  title = {{CAPTOR: text-encoder-free grounding by counting caption co-occurrence over the visual words of a frozen encoder}},
  year = {2026},
  eprint = {2611.05599},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05599},
}
Figure