Cross-modal Representation Learning and Relation Reasoning for Bidirectional Adaptive Manipulation

Lei Li, Kai Fan, Chun Yuan

Abstract

This paper introduces Cross-modal Adaptive Manipulation (CAM), a novel task that performs bidirectional editing of images and text simultaneously. By using a shared graph-based reasoning framework, the model can update an image to match a description while also rewriting the description to reflect the visual content. Leveraging Heterogeneous Representation Learning (HRL) with bridge nodes and Cross-modal Relation Reasoning (CRR) with graph masking, the model achieves state-of-the-art results on fine-grained attribute and relational manipulation benchmarks.

Introduction: Beyond One-Way Manipulation

Traditional controllable manipulation is usually a "one-way street." Existing systems typically focus on either text-to-image manipulation (editing an image based on a caption) or image-to-text manipulation (editing a caption based on an image).

Comparison of traditional manipulation tasks versus the proposed Cross-modal Adaptive Manipulation (CAM).

The authors identify three major gaps in traditional approaches:

  1. Lack of Symmetry: Models focus on a single output modality, ignoring the potential for mutual adaptation.
  2. Heavy Supervision Requirements: Single-modal tools often require expert-designed rules or specific priors.
  3. Context Loss: Models struggle to preserve independent information (like background) when manipulating specific attributes.

CAM enables "mutual supervision," where the image and text act as constraints for each other, reducing the need for manual effort and specific priors.

Methodology

The proposed framework consists of two parallel pipelines—a GAN for images and an Encoder-Decoder for text—linked by two core modules that facilitate cross-modal exchange.

The overall architecture of the CAM framework, featuring the HRL and CRR modules connecting the GAN and Encoder-Decoder pipelines.

Heterogeneous Representation Learning (HRL)

The model constructs an Image Graph (objects as nodes) and a Sentence Graph (words as nodes). The core innovation here is the use of Bridge Nodes. The model identifies objects that appear in both modalities (e.g., the word "bird" and the visual object "bird") and uses them as anchors to exchange information between the two graphs. Graph Convolutional Networks (GCNs) then propagate features across these extended, multi-modal graphs.

Cross-modal Relation Reasoning (CRR)

This module employs a bidirectional graph masking mechanism. It predicts "soft masks" to identify which nodes in the image graph are mismatched with the text, and vice versa. By masking "old" attributes and highlighting "new" ones, the model can surgically swap features—such as changing a bird's "red breast" to "blue"—without destroying the overall structure of the image or sentence.

Experimental Results

The model was evaluated on two benchmark datasets: CUB (Caltech-UCSD Birds-200-2011) for fine-grained attribute manipulation and Visual Genome (VG) for complex relation manipulation.

Key Performance Metrics

  • Image Quality (CUB): The model achieved an Inception Score (IS) of 8.34 and a Fréchet Inception Distance (FID) of 8.74, significantly outperforming previous state-of-the-art models like ManiGAN.
  • Relational Accuracy (Visual Genome): In tasks involving object removal or repositioning, the model showed lower reconstruction errors (MAE of 7.41 vs. 8.53 for SIMSG).
  • Text Generation: On the CUB dataset, the model produced highly accurate descriptions of manipulated images, achieving a BLEU score of 36.1.
  • Efficiency: Despite the dual-task nature, the model maintains an inference time of just 0.35 seconds for 100 images on a single RTX 3080.

Conclusion and Future Work

The CAM framework demonstrates that mutual supervision is highly efficient; linguistic structures help guide visual manipulation, while spatial layouts ground text generation. Focusing on co-occurring "bridge" objects provides a stable anchor for feature exchange that avoids the noise of global alignment.

Future work may address the current dependency on bridge nodes (which requires overlapping concepts) and explore extending the CAM framework to other modalities such as audio or video. While highly effective on standard benchmarks, extremely cluttered scenes with dozens of interacting objects remain a challenge for current graph-based reasoning approaches.

Summary updated

Abstract

Since single-modal controllable manipulation typically requires supervision of information from other modalities or cooperation with complex software and experts, this paper addresses the problem of cross-modal adaptive manipulation (CAM). The novel task performs cross-modal semantic alignment from mutual supervision and implements bidirectional exchange of attributes, relations, or objects in parallel, benefiting both modalities while significantly reducing manual effort. We introduce a robust solution for CAM, which includes two essential modules, namely Heterogeneous Representation Learning (HRL) and Cross-modal Relation Reasoning (CRR). The former is designed to perform representation learning for cross-modal semantic alignment on heterogeneous graph nodes. The latter is adopted to identify and exchange the focused attributes, relations, or objects in both modalities. Our method produces pleasing cross-modal outputs on CUB and Visual Genome.
BibTeX
@inproceedings{li2022cross,
  author = {Lei Li and Kai Fan and Chun Yuan},
  title = {{Cross-modal Representation Learning and Relation Reasoning for Bidirectional Adaptive Manipulation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/447},
}
Figure