ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Training

Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodolà, Francesco Locatello

Abstract

ASIF (Aligning Samples via Implicit Features) is a simple, non-parametric method that transforms separate, pre-trained image and text encoders into a joint multimodal system without any additional training. By utilizing a relatively small set of image-text pairs as a "Rosetta Stone," ASIF achieves zero-shot performance competitive with state-of-the-art models like CLIP and LiT, while being 250x more data-efficient and updatable in seconds.

Introduction and Problem Statement

Foundation multimodal models, such as CLIP, have revolutionized computer vision by enabling models to understand images through natural language. However, these models present significant hurdles: they require massive computational resources to train from scratch on hundreds of millions of pairs, they lack interpretability as "black boxes," and they are inflexible to updates without expensive retraining.

Comparison of training paradigms: CLIP (trains both), LiT (trains text), and ASIF (freezes both).

ASIF addresses these challenges by asking: Can we align vision and language spaces using existing, frozen unimodal models without training a single new neuron?

Methodology: Relative Representations

The core innovation of ASIF is the use of Relative Representations. Instead of relying on the absolute feature vectors produced by encoders, ASIF represents a data point by its similarity to a set of "anchors"—a multimodal dataset of image-text pairs.

The ASIF Mechanism

  1. Frozen Encoders: The system uses a frozen, pre-trained image encoder (e.g., DINO) and a frozen text encoder (e.g., Sentence-BERT).
  2. Anchor Set: A collection of image-text pairs serves as the reference. All images and texts are encoded separately and stored.
  3. Relative Mapping: To classify a new image, the system calculates its similarity to every image in the anchor set, creating a "similarity vector."
  4. The "As-If" Assumption: ASIF assumes that if a new image is similar to specific images in the anchor set, its ideal caption should be "similarly similar" to the corresponding texts in that set.
  5. Refinement: To reduce noise, the authors apply Sparsification (keeping only top-$k$ matches) and Exponentiation to emphasize the strongest semantic signals.
ASIF maps unimodal embeddings into a shared space by leveraging the relative distances to anchor points.

Because no weights are updated, the alignment happens entirely through the structure of the data itself, making the model inherently interpretable and instantly updatable.

Experimental Results

Zero-Shot Performance

Using only 1.6 million pairs from the Conceptual Captions 12M (CC12M) dataset—a fraction of the 400M used by CLIP—ASIF achieved 60.9% accuracy on ImageNet. This significantly outperforms CLIP models trained on similar small-scale data (31.3%) and approaches the performance of models like LiT (66.9%) that require active training of one encoder.

Instant Domain Adaptation

ASIF demonstrates remarkable flexibility in domain adaptation. On the EuroSAT satellite imagery dataset, a standard off-the-shelf ASIF model initially achieved 29.4% accuracy. By simply adding 100 satellite image-text pairs to the anchor set, the accuracy jumped to 82.2% instantly, without any gradient updates.

Key Insights

  • Interpretability by Design: Every dimension in an ASIF representation corresponds to a real image-text pair. If a model classifies an image as a "dog," users can inspect the specific anchor pairs that drove the decision.
  • Data-Centric Editing: Knowledge can be added or removed in seconds. Deleting embeddings from the anchor set effectively "unlearns" specific data, while adding new pairs expands the model's vocabulary.
  • Scaling Laws: ASIF follows scaling laws similar to trained models; performance improves predictably as the size of the anchor set increases, despite the lack of traditional training.

Limitations and Future Work

While powerful, ASIF's non-parametric nature introduces trade-offs. The memory footprint grows with the anchor set, requiring efficient storage like product quantization. Additionally, inference complexity increases linearly with the number of anchors, though vector databases (e.g., FAISS) can mitigate this. Finally, while excellent for retrieval and classification, it remains to be seen how this approach can be adapted for generative tasks like text-to-image synthesis.

Summary updated

BibTeX
@inproceedings{norelli2023asif,
  author = {Antonio Norelli and Marco Fumero and Valentino Maiorca and Luca Moschella and Emanuele Rodolà and Francesco Locatello},
  title = {{ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Training}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=XjOj3ZmWEl},
}
Figure