ReCoCo: From Patches to Compositions for Visual Representation Learning

Juan Gutiérrez, Saman Zadtootaghaj, José Luis Blanco-Murillo, Sara AtitoarXiv 2026 · arXiv

TL;DR

ReCoCo (Regional Composition Consistency) is a self-supervised learning objective that aligns the "composition" of shared regions between different image views. By matching the average distribution of visual prototypes rather than individual patches, it significantly boosts the quality of dense features for tasks like segmentation and clustering without sacrificing global image-level performance.

What Problem Does This Paper Address?

Self-supervised learning (SSL) has revolutionized how we train vision models without labels. Most successful methods, like DINO, focus on global consistency: ensuring that two different crops of the same image produce similar global representations. While this is great for image classification, it often fails to capture the fine-grained, "dense" features needed for tasks like semantic segmentation.

Existing "dense" SSL methods try to fix this by matching individual patches across views. However, this is difficult because different crops might have different scales or resolutions, making exact patch-to-patch correspondence unreliable. Other methods try to group patches into semantic regions first, but this introduces extra complexity and assumptions. ReCoCo seeks a middle ground: a way to enforce local consistency that is robust to scale changes and doesn't require explicit patch matching.

Methods

ReCoCo introduces Regional Composition Consistency. Instead of trying to find which patch in View A matches which patch in View B, the authors identify the overlapping region shared by both views.

Overview of ReCoCo's regional composition consistency.

The key steps are:

  1. Identify the Overlap: For any two augmented views (student and teacher), determine the spatial region they share.
  2. Compute Compositions: Within this shared region, the model predicts a probability distribution over a set of learned "visual prototypes" for every patch. These distributions are then averaged to create a single "composition" vector for the region.
  3. Align Compositions: The student is trained to match the teacher's regional composition using a cross-entropy loss.

A crucial design choice is averaging predictions rather than embeddings. As shown in the figure below, averaging embeddings can result in a "mean" vector that doesn't represent any actual visual concept. In contrast, averaging probability distributions preserves the "peaks" of multiple concepts (e.g., a region containing both a zebra and a giraffe), allowing the model to learn from complex regions without needing to segment them first.

Figure 1. ReCoCo matches regional compositions across views. (a) The student and teacher process two overlapping views of an image. Rather than matching individual patches, ReCoCo averages the patch predictions within the shared region to obtain one composition per view, then aligns the two compositions. Unlike averaging embeddings, averaging predictions preserves the contributions of multiple visual concepts (bottom). (b) Mean mIoU of six frozen segmentation settings (k-NN and linear probes on ADE20K, PASCAL VOC and Cityscapes) across backbones. (c) The same mean against ImageNet-1K linear accuracy (ViT-S/16). ∗Pretrained on LVD-142M [1].

Datasets

The models (ViT-S, ViT-B, and ViT-L) were pretrained on ImageNet-1K. Evaluation spanned a wide variety of downstream tasks and datasets:

  • Semantic Segmentation: ADE20K, PASCAL VOC, and Cityscapes.
  • Clustering: PASCAL VOC and COCO-Things.
  • Multi-label Classification: PASCAL VOC, MS-COCO, and Visual Genome.
  • Object/Region Retrieval: MS-COCO and PASCAL VOC.

Key Insights and Ideas

  • Correspondence-Free Alignment: By matching regional averages, ReCoCo avoids the "noisy" process of patch-to-patch matching. It doesn't matter if one view has 10 patches in the overlap and the other has 50; their aggregate "composition" should still be the same.
  • Preserving Multi-Concept Information: Averaging in the probability space acts like a histogram of visual concepts. This allows the model to supervise regions containing multiple objects naturally.
  • Feature Reorganization: Analysis shows that ReCoCo reduces feature variance within objects while increasing it between classes. This makes the features much more "compact" and easier for simple classifiers (like k-NN) to use.
  • No Extra Parameters: ReCoCo integrates into existing teacher-student frameworks (like iBOT) without adding any new learnable parameters or complex architectural modules.

Key Results

ReCoCo achieves state-of-the-art results for dense tasks among models trained on ImageNet-1K:

  • Segmentation: On PASCAL VOC, ReCoCo (ViT-S) improves over the iBOT baseline by a massive 14.6 mIoU points (from 49.7 to 64.3) using a simple k-NN probe.
  • Clustering: It outperforms DINOv2 on PASCAL VOC clustering mIoU (18.7 vs 17.1), despite DINOv2 being trained on the much larger LVD-142M dataset.
  • Retrieval: The method nearly doubles iBOT's performance in object retrieval on COCO, moving from 20.9 to 38.4 mAP.
  • Global Performance: Unlike many dense SSL methods that hurt image-level accuracy, ReCoCo maintains or slightly improves ImageNet-1K linear probe accuracy.

Limitations and Future Work

The authors note that because ReCoCo discards spatial ordering within the shared region to achieve correspondence-free matching, it is less effective for geometry-sensitive tasks.

Figure 3. Feature analysis with frozen ViT-S/16 features. (a) Variance of the patch features of VOC validation objects (mean squared cosine distance to the mean) with standard deviation. (b) SPair-71k [email protected] and depth RMSE.

As seen in Figure 3b (above), while ReCoCo excels at semantic tasks, it shows a slight degradation in keypoint correspondence (SPair) and no improvement in depth estimation (NYU/KITTI). Future work could involve finding ways to re-introduce spatial awareness into the regional objective without losing the benefits of the composition-based approach.

Summary updated

Abstract

Self-supervised learning has produced powerful visual representations, yet learning strong dense features remains challenging. While consistency between augmented views provides strong image-level supervision, extending this principle to local visual content is less straightforward. Existing approaches often match individual patches across views or identify coherent regions, introducing uncertainties in correspondence estimation and assumptions about how visual content should be grouped. We introduce ReCoCo (Regional Composition Consistency), a principled approach that extends cross-view consistency to the regional level without requiring such correspondences or semantic grouping. ReCoCo represents the content shared between augmented views through compositions of their patch-level predictions over learned visual prototypes. By aligning these compositions, the model learns from regions containing multiple visual concepts without requiring them to be separated beforehand. ReCoCo integrates naturally with teacher-student self-distillation and introduces no additional learned parameters. Extensive experiments demonstrate substantial improvements in frozen-feature segmentation, clustering, and regional retrieval while maintaining competitive image-level recognition, alleviating the trade-off between dense and global representation quality.
BibTeX
@misc{gutierrez2026recoco,
  author = {Juan Gutiérrez and Saman Zadtootaghaj and José Luis Blanco-Murillo and Sara Atito},
  title = {{ReCoCo: From Patches to Compositions for Visual Representation Learning}},
  year = {2026},
  eprint = {2611.24806},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24806},
}
Figure