Reason from Context with Self-supervised Learning

Xiao Liu, Ankur Sikarwar, Joo Hwee Lim, Gabriel Kreiman, Zenglin Shi, Mengmi ZhangarXiv 2022 · OpenReview

Abstract

Traditional computer vision models often prioritize local object features over global contextual relationships. In contrast, human perception relies heavily on contextual priors to identify objects in ambiguous or occluded settings. This paper introduces SeCo (Self-supervised Context reasoning), a novel computational framework that utilizes a biologically inspired two-stream architecture and an external memory module to learn contextual associations without human-labeled data. Through a series of psychophysics experiments and benchmarking on datasets like FRINE and COCO-OCD, we demonstrate that SeCo not only outperforms existing self-supervised learning (SSL) models but also exhibits high alignment with human behavioral patterns in scene understanding tasks.

Introduction

When observing a kitchen, a human does not need to see every pixel of a fork to identify it; its proximity to a plate and a napkin provides sufficient evidence. This ability to "see the elephant in the room" through contextual reasoning is a hallmark of human intelligence. However, most contemporary AI models are trained on isolated object recognition, requiring millions of labeled examples to learn relationships that humans acquire through passive observation.

Current Self-Supervised Learning (SSL) methods, such as SimCLR or DINO, have made significant strides in learning robust feature representations. Yet, they often fail to capture the complex spatial and semantic dependencies between objects in a scene. This research addresses two primary gaps: the mechanism by which these relationships are learned without explicit supervision and the architectural requirements for reasoning about object-context interactions.

The SeCo Framework

The researchers developed SeCo, a model designed to mimic the dual-pathway processing of the human brain.

Biologically Inspired Architecture

SeCo utilizes a Two-Stream Architecture that mirrors the ventral and dorsal pathways in the visual cortex. One stream focuses on the target object's identity, while the other processes the surrounding scene context. Central to this architecture is an External Memory Module, inspired by the hippocampus. This module stores "latent contextual priors"—representations of typical object-context pairings learned during training. When presented with a scene, the model queries this memory to retrieve a representation of the expected object.

Learning Objective

To avoid the need for manual labels, SeCo employs a self-supervised objective. It uses a joint loss function based on the VICReg (Variance-Invariance-Covariance Regularization) framework. This approach ensures that the model learns informative representations while preventing "representation collapse," where the model might otherwise output identical vectors for different inputs.

Experimental Methodology

The performance of SeCo was evaluated against both state-of-the-art AI models and human participants across several tasks.

Psychophysics and Tasks

  1. Lift-the-Flap Task: Participants and models were shown scenes where a central object was obscured by a black patch. They were required to identify the hidden object based solely on the surrounding context.
  2. Object Priming: This task measured spatial reasoning by asking participants to indicate where a specific object should be placed within a given scene.

Benchmarks and Datasets

The study utilized a diverse set of datasets to test generalization:

  • FRINE (Fribble In the sceNE): A synthetic benchmark using novel "fribble" objects in virtual environments to control for prior human knowledge.
  • COCO-OCD & COCO-VOC: Real-world datasets curated to test context reasoning and domain transfer.
  • HOP (Human Object Priming): A dataset of human-generated heatmaps used to validate the "human-likeness" of model predictions.

Key Findings

The results indicate that SeCo represents a significant advancement in contextual AI.

1. Superior Accuracy and Human Alignment: SeCo outperformed SSL baselines like DINO and SimCLR on the lift-the-flap task. More importantly, its error patterns showed a strong correlation with human behavior ($r \geq 0.8$), suggesting that the model utilizes similar logic to human observers.

2. The Role of Memory: Ablation studies revealed that the external memory module is critical; its removal resulted in a 12.5% decrease in accuracy. This confirms that storing "prototypes" of scenes is essential for filling in missing information.

3. Robustness to Visual Degradation: Both humans and SeCo maintained high performance even when the context was heavily blurred. This suggests that contextual reasoning relies more on the "global gist" of a scene than on high-frequency visual details.

4. Zero-Shot Generalization: SeCo demonstrated an impressive ability to transfer knowledge across domains—for instance, from natural images to synthetic environments—outperforming many supervised models in zero-shot scenarios.

Discussion and Future Directions

While SeCo bridges a vital gap, several limitations remain. The current memory module is static, whereas biological memory is dynamic and constantly consolidated. Furthermore, SeCo relies on pre-defined object proposals, whereas humans use active attention and eye movements to scan scenes. Future work will explore incorporating recurrent dynamics and causal reasoning—such as understanding physical constraints like gravity—to further refine the model's "common sense."

By integrating principles from cognitive science with self-supervised learning, SeCo provides a robust foundation for AI that understands the world as a coherent, interconnected scene rather than a collection of isolated pixels.

Summary updated

BibTeX
@misc{liu2022reason,
  author = {Xiao Liu and Ankur Sikarwar and Joo Hwee Lim and Gabriel Kreiman and Zenglin Shi and Mengmi Zhang},
  title = {{Reason from Context with Self-supervised Learning}},
  year = {2022},
  url = {https://openreview.net/forum?id=HDbd4Z6rPlX},
}
Figure