Recursive CLIP: Vision-Language Distillation with Looped Transformer Blocks

Szymon Mazurek, Alberto Presta, Michał ByraarXiv 2026 · arXiv

TL;DR

Recursive CLIP introduces "looped" transformer blocks to vision-language models, reusing a small set of parameters multiple times to maintain computational depth while drastically reducing model size. By combining this architecture with knowledge distillation, the authors show that CLIP models can shed up to 75% of their parameters while remaining highly effective for tasks like zero-shot classification, visual grounding, and few-shot learning.

What Problem Does This Paper Address?

State-of-the-art vision-language models like CLIP typically use "deep" architectures (like ViT-B/16) where every transformer block has its own unique set of parameters. While this provides massive representational capacity, it leads to a high memory footprint, making deployment on edge devices difficult.

The central question is: Do we really need unique parameters at every layer? Previous work in NLP (like ALBERT) suggested that sharing weights across layers can work. This paper investigates if this "recurrence" can be applied to the dual-encoder setup of CLIP without sacrificing the transferable features that make CLIP so useful.

Methods

The authors propose two model families:

  • rCLIP: Only the vision encoder uses recursive blocks.
  • r²CLIP: Both the vision and text encoders use recursive blocks.

The core idea is simple: instead of $L$ unique blocks, they use $N$ distinct blocks and execute them $M$ times, such that $N \times M = L$. For a standard ViT-B (where $L=12$), they test configurations like $1b12i$ (1 block applied 12 times) or $6b2i$ (6 blocks applied twice). This keeps the total FLOPs (computational cost) identical to a standard ViT-B but reduces the stored parameters.

Figure 2. Overview of the experimental approach. (1) A recursive encoder replaces L distinct transformer blocks (A) with N blocks applied M times, N·M = L (B), keeping computational depth fixed while reducing stored parameters. (2) Recursive (rCLIP, r²CLIP) and non-recursive students are distilled from a frozen DFN-2B ViT-B/16 teacher on CC3M + CC12M. (3) The resulting models are evaluated on classification (zero- and few-shot adapted), cross-modal retrieval, visual grounding and concept unlearning.

To train these models effectively, they use Knowledge Distillation (KD). They take a powerful "Teacher" model (a ViT-B/16 trained on the massive DFN-2B dataset) and force the smaller recursive "Student" models to mimic its embeddings using several loss functions:

  • Feature Distillation (FD): Matching final embeddings.
  • Contrastive Relational Distillation (CRD): Matching the similarity structure of a batch.
  • Interactive Contrastive Learning (ICL): Contrasting student features against teacher features.

Datasets

  • Training (Distillation): A combination of CC3M and CC12M (roughly 14 million image-caption pairs).
  • Teacher Pre-training: DFN-2B (2 billion pairs).
  • Evaluation:
    • Zero-shot Classification: ImageNet-1k and its variants (V2, Rendition, Sketch).
  • Retrieval: MSCOCO and Flickr30k.
  • Visual Grounding: RefCOCO, RefCOCO+, RefCOCOg, ReferItGame, and Flickr30k Entities.
  • Adaptation/Unlearning: Flowers102, OxfordPets, and Food101.

Key Insights and Ideas

  • The "Saturation" Point: Performance improves as you add more unique blocks ($N$), but the gains saturate quickly. Moving from 1 unique block to 2 provides a massive boost, but moving from 4 to 6 provides diminishing returns.
  • Text vs. Vision Robustness: The text encoder is surprisingly resilient to weight sharing. In many cases, an $r^2CLIP$ model (recursive text) performs almost as well as an $rCLIP$ model (unique text), despite having significantly fewer parameters.
  • Activation Matching (AM) Fails: One common distillation technique is to make the student's intermediate layers match the teacher's. This failed for recursive models. The authors hypothesize that a looped block follows a different "trajectory" in representation space than a sequence of unique blocks, making layer-to-layer matching an overly restrictive constraint.
  • Adapter Efficiency: When using LoRA (Low-Rank Adaptation) for downstream tasks, the adapters are attached to the unique blocks. Because the blocks are reused, the adapters are also reused. This means recursive models require an order of magnitude fewer parameters to fine-tune than standard models.

Key Results

The trade-off between accuracy and size is the headline result. As seen in Figure 1, recursive models provide a smooth "menu" of options for developers.

Accuracy vs. Parameter count trade-off
  • Parameter Reduction: The most compressed model, r²CLIP 1b12i, uses only 37M parameters (down from ~150M in a standard ViT-B) while maintaining a zero-shot ImageNet accuracy of 52% (when distilled).
  • Closing the Gap: The rCLIP 6b2i model reaches 68.8% ImageNet accuracy, coming within 4 percentage points of the non-recursive reference distilled on the same data.
  • Grounding Excellence: Surprisingly, for visual grounding tasks, the recursive models performed almost identically to the non-recursive baselines, suggesting that spatial reasoning in CLIP doesn't require layer-specific parameterization.

Limitations and Future Work

  • Fixed Iterations: The authors used a fixed $N \times M = 12$ grid. Future work could explore "early exiting" or varying the number of iterations per block based on the input complexity.
  • Hyperparameter Tuning: The training recipe was largely adopted from standard non-recursive distillation papers. It is possible that recursive architectures require specific learning rate schedules or different weight decay settings to reach their full potential.
  • Scale: This study focused on the ViT-B scale. Whether these insights hold for ViT-Large or Giant models remains to be seen.

Summary updated

Abstract

Contrastive vision-language models such as CLIP typically rely on independently parameterized transformer blocks across depth. We investigate the extent to which these blocks can instead be reused recurrently while preserving the transferable capabilities of CLIP. We study recursive transformer backbones in which a small set of $N$ transformer blocks is applied $M$ times per forward pass. By holding $N \cdot M$ fixed to the depth of a standard ViT-B, our models preserve computational depth while progressively increasing parameter sharing. We consider two recursive CLIP model families: rCLIP, where only the vision encoder is recursive, and r$^2$CLIP, where recurrence is also introduced in the text encoder. We study these models under both direct contrastive training and through knowledge distillation from a strong ViT-B/16 teacher pretrained on DFN-2B, using the substantially smaller CC3M and CC12M datasets. We evaluate the resulting models on zero-shot classification and cross-modal retrieval, and further study their adaptability through few-shot learning, visual grounding, and concept unlearning. Our results show that CLIP can tolerate substantial recurrent block reuse, although its effect depends on the encoder and downstream task. Recurrence also provides parameter efficiency by construction, since fewer distinct transformer blocks need to be stored and adapted. Our results establish recurrent block reuse as a practical design space for vision-language models and highlight its implications for knowledge transfer, downstream adaptation, and parameter efficiency.
BibTeX
@misc{mazurek2026recursive,
  author = {Szymon Mazurek and Alberto Presta and Michał Byra},
  title = {{Recursive CLIP: Vision-Language Distillation with Looped Transformer Blocks}},
  year = {2026},
  eprint = {2610.34563},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34563},
}
Figure