TL;DR
CAPTOR provides a high-performance text interface for "language-blind" frozen image encoders without requiring a text encoder or gradient-based training. By simply counting how terms co-occur with "visual words" in a captioned dataset, it achieves state-of-the-art training-free grounding on natural images and significantly outperforms specialized pathology models that were actually trained with language.
What Problem Does This Paper Address?
Modern vision-language models like CLIP rely on a text encoder perfectly aligned with an image encoder. However, in many specialized fields—most notably computational pathology—the best image encoders are often trained on massive unlabeled datasets (e.g., millions of tissue slides) and lack a corresponding text encoder.
To "talk" to these models, researchers usually have to train a new alignment, which is computationally expensive, requires large paired datasets, and must be repeated for every new encoder. The authors of CAPTOR ask a provocative question: Can we ground text in images using only a frozen image encoder and a pile of captioned images, without ever using a text encoder or backpropagation?
Methods
CAPTOR (Caption-Anchored Patch-Term Occurrence Representation) operates in two distinct stages:
- Offline Preparation:
- Visual Words: The system takes a frozen image encoder and runs k-means clustering on patch tokens from a caption corpus to create $K$ "visual words."
- Term Signatures: For every term in the vocabulary, it calculates a signature—essentially a standardized measure of how much more (or less) a specific visual word appears in images where the term is mentioned in the caption versus images where it isn't.
- Frequency Correction: To ensure rare terms don't dominate the scores, CAPTOR uses a "permutation null" (shuffling captions) to put all term signatures on a common, comparable scale.
- Online Inference:
- When a new image arrives, it is passed through the frozen encoder once. Each patch is assigned to its nearest visual word (forming an image factor $\Phi$).
- Grounding a query term is then reduced to a simple table lookup in the pre-computed signature table ($S$). The final compatibility matrix is factorized as $A = S\Phi$.
Datasets
- Natural Images: The authors used COCO (15k-30k training captions) to build the signature table and evaluated on COCO-validation, Pascal VOC 2012, and Cityscapes.
- Computational Pathology: The table was built using PathCap (32k captioned figures). Evaluation was performed on specialized benchmarks: BCSS (breast cancer) and LUAD-HistoSeg (lung cancer).
- Naming Benchmarks: PanNuke and NCT-CRC were used to test how well the system could name visual words across different organs.
Key Insights and Ideas
- Counting is a Powerful Proxy for Alignment: The core "aha moment" is that co-occurrence statistics between captions and visual clusters can replace the complex latent-space alignment of neural text encoders.
- The Importance of the "Null": Raw co-occurrence counts are biased toward rare words. By dividing signatures by a permutation null (Figure 2), CAPTOR ensures that "person" and "handbag" can be compared fairly, even though one is vastly more common than the other.
- Factorization for Efficiency: Because the matrix factorizes, an image is encoded exactly once. Adding a new query term doesn't require a new pass through a text encoder; it's just adding a row to a small table.
- Domain Superiority: In specialized domains like pathology, general-purpose text encoders (like CLIP's) fail because they don't know the vocabulary. CAPTOR learns the domain-specific vocabulary directly from the captions of that domain.
Key Results
- Natural Images: On COCO, CAPTOR reached 0.565 mIoU, significantly beating training-free baselines like ProxyCLIP (0.502). It performed on par with Talk2DINO, despite Talk2DINO using a text encoder and five times more caption data.
- Pathology Breakthrough: This is where CAPTOR shines. On breast cancer slides, CAPTOR using the UNI encoder (which has no native language support) outperformed the strongest baseline that did use a pathology-specific text encoder by 0.132 mIoU.
- Speed and Efficiency: Building the entire signature table takes only 4.3 seconds on a GPU (or ~1 minute on a CPU). Adding a new term to the vocabulary is roughly 30 times faster than a pass through CLIP’s text encoder.
Limitations and Future Work
- Vocabulary Constraints: CAPTOR can only ground terms that appear frequently enough in the caption corpus to clear the "permutation null."
- Bag-of-Words Limitations: The current method does not handle complex linguistic structures like negation ("no tumor"), attributes ("red car"), or spatial relations ("cat on a mat").
- Visual Word Resolution: The grounding is limited by the quality of the initial k-means clustering. If the frozen encoder doesn't visually distinguish between two concepts, CAPTOR cannot separate them.
- Context Absorption: The authors noted that objects often seen in specific settings (like skis on snow) can "absorb" the signatures of their surroundings, leading to some background pixels being mislabeled as the object.
Summary updated