TL;DR
ReCoCo (Regional Composition Consistency) is a self-supervised learning objective that aligns the "composition" of shared regions between different image views. By matching the average distribution of visual prototypes rather than individual patches, it significantly boosts the quality of dense features for tasks like segmentation and clustering without sacrificing global image-level performance.
What Problem Does This Paper Address?
Self-supervised learning (SSL) has revolutionized how we train vision models without labels. Most successful methods, like DINO, focus on global consistency: ensuring that two different crops of the same image produce similar global representations. While this is great for image classification, it often fails to capture the fine-grained, "dense" features needed for tasks like semantic segmentation.
Existing "dense" SSL methods try to fix this by matching individual patches across views. However, this is difficult because different crops might have different scales or resolutions, making exact patch-to-patch correspondence unreliable. Other methods try to group patches into semantic regions first, but this introduces extra complexity and assumptions. ReCoCo seeks a middle ground: a way to enforce local consistency that is robust to scale changes and doesn't require explicit patch matching.
Methods
ReCoCo introduces Regional Composition Consistency. Instead of trying to find which patch in View A matches which patch in View B, the authors identify the overlapping region shared by both views.
The key steps are:
- Identify the Overlap: For any two augmented views (student and teacher), determine the spatial region they share.
- Compute Compositions: Within this shared region, the model predicts a probability distribution over a set of learned "visual prototypes" for every patch. These distributions are then averaged to create a single "composition" vector for the region.
- Align Compositions: The student is trained to match the teacher's regional composition using a cross-entropy loss.
A crucial design choice is averaging predictions rather than embeddings. As shown in the figure below, averaging embeddings can result in a "mean" vector that doesn't represent any actual visual concept. In contrast, averaging probability distributions preserves the "peaks" of multiple concepts (e.g., a region containing both a zebra and a giraffe), allowing the model to learn from complex regions without needing to segment them first.
Datasets
The models (ViT-S, ViT-B, and ViT-L) were pretrained on ImageNet-1K. Evaluation spanned a wide variety of downstream tasks and datasets:
- Semantic Segmentation: ADE20K, PASCAL VOC, and Cityscapes.
- Clustering: PASCAL VOC and COCO-Things.
- Multi-label Classification: PASCAL VOC, MS-COCO, and Visual Genome.
- Object/Region Retrieval: MS-COCO and PASCAL VOC.
Key Insights and Ideas
- Correspondence-Free Alignment: By matching regional averages, ReCoCo avoids the "noisy" process of patch-to-patch matching. It doesn't matter if one view has 10 patches in the overlap and the other has 50; their aggregate "composition" should still be the same.
- Preserving Multi-Concept Information: Averaging in the probability space acts like a histogram of visual concepts. This allows the model to supervise regions containing multiple objects naturally.
- Feature Reorganization: Analysis shows that ReCoCo reduces feature variance within objects while increasing it between classes. This makes the features much more "compact" and easier for simple classifiers (like k-NN) to use.
- No Extra Parameters: ReCoCo integrates into existing teacher-student frameworks (like iBOT) without adding any new learnable parameters or complex architectural modules.
Key Results
ReCoCo achieves state-of-the-art results for dense tasks among models trained on ImageNet-1K:
- Segmentation: On PASCAL VOC, ReCoCo (ViT-S) improves over the iBOT baseline by a massive 14.6 mIoU points (from 49.7 to 64.3) using a simple k-NN probe.
- Clustering: It outperforms DINOv2 on PASCAL VOC clustering mIoU (18.7 vs 17.1), despite DINOv2 being trained on the much larger LVD-142M dataset.
- Retrieval: The method nearly doubles iBOT's performance in object retrieval on COCO, moving from 20.9 to 38.4 mAP.
- Global Performance: Unlike many dense SSL methods that hurt image-level accuracy, ReCoCo maintains or slightly improves ImageNet-1K linear probe accuracy.
Limitations and Future Work
The authors note that because ReCoCo discards spatial ordering within the shared region to achieve correspondence-free matching, it is less effective for geometry-sensitive tasks.
As seen in Figure 3b (above), while ReCoCo excels at semantic tasks, it shows a slight degradation in keypoint correspondence (SPair) and no improvement in depth estimation (NYU/KITTI). Future work could involve finding ways to re-introduce spatial awareness into the regional objective without losing the benefits of the composition-based approach.
Summary updated