Towards Learning Implicit Symbolic Representation for Visual Reasoning

Chen Sun, Calvin Luo, Xingyi Zhou, Anurag Arnab, Cordelia SchmidICLR 2023 · OpenReview

TL;DR

This paper introduces a self-supervised learning framework called Implicit Symbolic Concept Learner (IS-CL) that enables end-to-end visual reasoning directly from raw video inputs. By training a Transformer network to reconstruct video frames from compressed "slot" tokens, IS-CL learns implicit symbolic representations without requiring explicit object detection or predefined symbolic structures, achieving competitive or better performance than existing neuro-symbolic methods.

What Problem Does This Paper Address?

Visual reasoning tasks, which involve understanding causality, object interactions, and temporal dynamics from visual data, are notoriously challenging. A common belief is that to achieve robust generalization in these tasks, especially compositional generalization (i.e., combining known concepts in novel ways), an explicit, structured representation of the visual scene is necessary. This often involves complex pipelines like object detection followed by symbolic reasoning or specialized neural networks. However, creating these explicit representations is difficult because the ideal symbolic vocabulary can vary significantly across tasks, and reliably extracting these symbols from raw pixels is a hard problem. This paper aims to bridge this gap by demonstrating that a general-purpose neural network, trained with self-supervision, can learn these implicit symbolic representations directly from raw visual inputs, enabling end-to-end visual reasoning.

Methods

The core of the proposed approach, IS-CL, lies in a self-supervised pre-training strategy. It consists of two main components:

  1. Image Encoder (Vision Transformer - ViT): Each video frame is processed by a ViT. Crucially, the ViT is augmented with special "slot" tokens that attend to the image patches. The goal is for these slot tokens to act as a compressed, representational bottleneck, capturing the essential information of the image.
  2. Temporal Transformer: This component takes the slot tokens (and unmasked patch tokens) from a sequence of frames and propagates temporal information.

The self-supervised pre-training objective is masked autoencoding (MAE) on unlabeled video data. The network is tasked with reconstructing masked patches of an image, using the compressed temporal context from other frames (via the slot tokens and patch tokens) as input. The hypothesis is that to minimize the reconstruction loss, the network must learn:

  • A compact representation of each image within its slot tokens.
  • To capture temporal dynamics and object permanence from the temporal context.

During transfer learning, the image decoder is discarded. The pre-trained image encoder and temporal transformer are then fine-tuned on downstream visual reasoning tasks.

Datasets

The paper evaluates its approach on two existing visual reasoning benchmarks:

  • CATER: A dataset for compositional actions and temporal reasoning, featuring videos of objects moving in a scene. The task is to predict the final location of a "snitch" object.
  • ACRE: A dataset designed to test abstract causal reasoning, inspired by the Blicket experiment. It involves inferring causal relationships between objects and their effects on a platform.

The pre-training is performed on unlabeled videos from the CATER dataset.

Key Insights and Ideas

  • Implicit Symbolic Representation is Achievable: The paper's central claim is that complex, symbolic-like representations can be learned implicitly by general-purpose neural networks (Transformers) through self-supervised learning, without explicit symbolic supervision.
  • Self-Supervised Pre-training is Crucial: The authors emphasize that self-supervised pre-training is essential for achieving compositional generalization. This suggests that the pre-text task of video reconstruction forces the network to learn the underlying structure and dynamics needed for reasoning.
  • Slot Tokens as Implicit Symbols: The "slot tokens" are hypothesized to act as learned, implicit symbolic representations. Their ability to capture object permanence and temporal dynamics is key to the framework's success.
  • End-to-End Learning Advantage: By avoiding a separate, explicit symbolic representation extraction stage, the proposed method allows for end-to-end learning, which can be more flexible and avoid information loss associated with fixed symbolic representations.
  • Transformer's Adaptability: The use of Transformers for both image encoding and temporal modeling demonstrates their versatility in learning complex spatio-temporal representations.

Key Results

  • The proposed IS-CL framework achieves competitive or better performance compared to recent neuro-symbolic approaches on both CATER and ACRE benchmarks.
  • Crucially, IS-CL achieves this performance without requiring explicit object-centric representation or object-level supervision, which is a common requirement for many existing methods.
  • Ablation studies show that the number of slot tokens is an important hyperparameter, with different optimal values for CATER (1 slot) and ACRE (higher number of slots, e.g., 8). This suggests that the complexity of the reasoning task influences the required representational capacity of the implicit symbols.
  • The paper demonstrates that IS-CL outperforms MAE and VideoMAE pre-training methods on CATER and ACRE by significant margins.

Limitations and Future Work

  • Hyperparameter Sensitivity: The paper notes that network inductive biases, such as the number of slot tokens, play an important role in transfer learning performance. Further research could explore more robust methods for determining these hyperparameters.
  • Probing Learned Representations: The authors suggest future work should focus on "probing experiments to understand the information encoded by the slot tokens." This would provide deeper insights into what these implicit symbols actually represent.
  • Scalability: While the current work focuses on specific visual reasoning benchmarks, future work could involve applying IS-CL to larger-scale natural image and video datasets to test its broader applicability and generalization capabilities.

Summary updated

Abstract

Visual reasoning tasks are designed to test a learning algorithm's capability to infer causal relationships, discover object interactions, and understand temporal dynamics, all from visual cues. It is commonly believed that to achieve compositional generalization on visual reasoning, an explicit abstraction of the visual scene must be constructed; for example, object detection can be applied to the visual input to produce representations that are then processed by a neural network or a neuro-symbolic framework. We demonstrate that a simple and general self-supervised approach is able to learn implicit symbolic representations with general-purpose neural networks, enabling the end-to-end learning of visual reasoning directly from raw visual inputs. Our proposed approach ``compresses'' each frame of a video into a small set of tokens with a transformer network. The self-supervised learning objective is to reconstruct each image based on the compressed temporal context. To minimize the reconstruction loss, the network must learn a compact representation for each image, as well as capture temporal dynamics and object permanence from temporal context. We evaluate the proposed approach on two visual reasoning benchmarks, CATER and ACRE. We observe that self-supervised pretraining is essential to achieve compositional generalization for our end-to-end trained neural network, and our proposed method achieves on par or better performance compared to recent neuro-symbolic approaches that often require additional object-level supervision.
BibTeX
@inproceedings{sun2023towards,
  author = {Chen Sun and Calvin Luo and Xingyi Zhou and Anurag Arnab and Cordelia Schmid},
  title = {{Towards Learning Implicit Symbolic Representation for Visual Reasoning}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=V8isglQkt74},
}
Figure