Introduction
OLIVE (Object Level In-Context Visual Embeddings) represents a paradigm shift in Vision-Language Models (VLMs). Instead of processing images as arbitrary grids of patches, OLIVE represents specific objects as single, compressed vectors. By integrating these object-level embeddings with a retrieval-augmented in-context learning approach, the framework enables Large Language Models (LLMs) to reason about rare or unseen visual concepts with high efficiency and minimal training.
The Limitations of Patch-Based Vision
Current VLMs, such as LLaVA, typically slice images into a grid of patches. This approach suffers from three primary drawbacks:
- Granularity Mismatch: Text tokens represent discrete concepts, whereas image patches are arbitrary spatial segments. Object features are often fragmented across multiple patches and mixed with background noise.
- Context Bloat: Processing hundreds of patch embeddings leads to extremely long input sequences, making multi-image in-context learning computationally prohibitive.
- The "Unseen" Problem: Models often fail or hallucinate when encountering rare objects or domain-specific data (e.g., medical imaging) underrepresented in their training sets.
Methodology: Object-Level Reasoning
OLIVE addresses these issues through a three-component architecture focused on object-level semantic representation.
1. Lightweight Object Encoder
Rather than passing all image patches, OLIVE utilizes a mask (from user input or a segmentation model like SAM) to isolate an object. A 20M-parameter transformer compresses these masked patches into a single visual vector.
2. Visual Object Retrieval
To handle novel concepts, the system maintains a retrieval set. For any query object, it identifies the $k$ most similar visual examples using cosine similarity.
3. Code-Switched In-Context Prompting
The LLM receives a prompt mixing text and retrieved visual vectors. The [obj] tokens are replaced directly by compressed visual embeddings, allowing the model to "see" examples of new concepts within the prompt, analogous to Retrieval-Augmented Generation (RAG) in text.
Evaluation and Key Insights
OLIVE was evaluated on standard datasets (COCO, refCOCOg) and specialized domains like ChestX-Ray8 (CXR8) for zero-shot adaptation.
Zero-Shot Domain Adaptation
On the CXR8 dataset, OLIVE-R (retrieval-only) achieved 33.5% accuracy, surpassing BioMedCLIP (32.5%), despite the latter being pre-trained on massive medical datasets. This demonstrates OLIVE's ability to adapt to specialized domains using only a small set of labeled examples.
Rapid Concept Adaptation
The generative model, OLIVE-RG, successfully describes animals unseen during training by leveraging retrieved examples.
Efficiency and Robustness
- Semantic Clustering: PCA analysis shows the encoder naturally clusters similar objects (e.g., dogs, fruits) without explicit category training.
- Background Immunity: By using precise masks, OLIVE is robust to "spurious features" that often confuse models relying on bounding boxes or full images.
- Compression: Representing objects as single vectors allows for dozens of in-context examples, maintaining a nearly flat context length compared to the steep linear increase seen in standard VLMs.
Conclusion and Future Directions
OLIVE provides a scalable and efficient path for VLMs to handle long-tail visual concepts. While highly effective, the model currently requires external masks for grounding and faces an information bottleneck by compressing complex objects into single vectors. Future work may explore "visually resampling" objects into multiple latent vectors to better balance compression with descriptive detail.
Summary updated