Disentanglement-Based Equivariant Learning for Compositional VQA

Zhou Du, Zhaoquan Yuan, Xiao Wu, Changsheng Xu

TL;DR

This paper introduces DEAL, a novel framework for compositional Visual Question Answering (VQA) that tackles the challenge of models failing to generalize to unseen combinations of concepts. DEAL achieves this by disentangling visual and linguistic concepts using causality-inspired interventions and then enforcing an equivariant constraint during learning, ensuring that transformations in input concepts lead to predictable transformations in output reasoning.

What Problem Does This Paper Address?

Compositional VQA is a critical task where models must understand and reason about novel combinations of concepts they've encountered during training. Existing VQA models often struggle with this due to:

  • Spurious Correlations: They learn superficial associations between concepts (e.g., "yellow" and "sphere") rather than true causal relationships, leading to errors when these associations don't hold.
  • Lack of Compositional Variation: Training often uses restricted concept combinations, preventing models from learning how to adapt to new arrangements.
  • Reliance on External Aids: Many state-of-the-art methods require additional training signals or pre-training, which isn't always feasible in real-world scenarios.

Humans, however, excel at this by disentangling individual concepts and understanding how their combinations affect reasoning. This paper aims to imbue VQA models with this capability.

Methods

The proposed DEAL framework has two core components:

  1. Disentanglement-Based Concept Learning:
    • Causality-Inspired Interventions: The authors leverage the principle of causal intervention. By selectively altering features related to one concept while keeping another invariant, they force the model to learn representations that are independent of each other. This is implemented within a re-encoding framework that includes encoders, decoders, and reconstruction objectives to ensure the learned representations are meaningful.
  2. Self-Supervised Learning: This disentanglement process is achieved without requiring explicit labels for concepts, relying instead on the structure of the data and the intervention mechanism.
  3. Equivariant Transformation:
    • Compositional Transformation: After disentangling concepts, the model learns to handle compositional variations. This involves transforming input concept representations (e.g., changing the color of a shape) and then enforcing an "equivariant constraint."
    • Equivariance: This constraint ensures that if the input concepts are transformed in a specific way, the model's output reasoning (represented by a global embedding) transforms predictably. This is achieved by measuring the transformation ratio and penalizing deviations from this expected output transformation.

Datasets

The paper uses the following datasets:

  • CLEVR-CoGenT: A diagnostic dataset designed for testing visual compositional generalization. It's a variant of CLEVR with synthetic 3D images and automatically generated questions.
  • GQA-SGL: A dataset derived from GQA, specifically designed to test linguistic compositional generalization. It uses real-world images and questions generated with ground-truth programs, featuring novel linguistic compositions.
  • CLEVR: The original version, used for evaluating performance in generic (in-distribution) VQA scenarios.
  • VQAv2.0: A widely used VQA benchmark, also used for evaluating in-distribution performance.

Key Insights and Ideas

  • Causality as a Foundation for Disentanglement: The paper argues that true concept disentanglement in VQA should be guided by causal principles, not just statistical independence. Interventions that mimic causal manipulation are key.
  • Equivariance as a Mechanism for Compositional Reasoning: The core idea is that if a model can predict how its reasoning should change when input concepts are altered, it can better handle novel combinations. Equivariance provides a principled way to enforce this.
  • End-to-End Learning with Ground-Truth Answers Only: A significant advantage is that DEAL only requires ground-truth answers for training, making it more broadly applicable than methods needing explicit concept labels or programs.
  • The Synergy of Disentanglement and Equivariance: The ablation studies strongly suggest that both disentanglement and equivariance are crucial and complementary for achieving strong compositional generalization.

Key Results

  • State-of-the-Art Performance: DEAL outperforms existing SOTA methods on both visual (CLEVR-CoGenT) and linguistic (GQA-SGL) compositional generalization tasks, achieving improvements of 1.5% and 2.4% respectively.
  • Reduced Generalization Gap: DEAL exhibits significantly smaller performance drops between in-distribution and generalization settings compared to other approaches, indicating more robust generalization. For instance, on CLEVR-CoGenT, the gap is 10.9% for DEAL, compared to much larger gaps for methods like MDETR (23.1%) and NS-VQA (35.9%).
  • Competitive In-Distribution Performance: Despite focusing on generalization, DEAL maintains competitive performance on standard VQA benchmarks like CLEVR and VQAv2.0.
  • Robustness to Hyperparameters: The model shows consistent performance improvements across a range of hyperparameter values for the disentanglement and equivariance losses.

Limitations and Future Work

  • Struggles with Counting and Multi-Step Reasoning: The paper acknowledges that DEAL, like many other models, still struggles with complex counting questions and tasks requiring intricate multi-step reasoning. This is attributed to the holistic nature of its Transformer-based architecture, which may not adapt as well to varying question lengths and types as modular approaches.
  • Integration with Modular Reasoning: The authors suggest future work could involve integrating their disentanglement and equivariant learning methods into modular reasoning frameworks. This could potentially combine the strengths of both approaches: disentangled representations and equivariant learning with the explicit step-by-step reasoning of modular networks.
  • Broader Applicability: DEAL's potential for integration with other multimodal models, such as VLMs and MLLMs, is highlighted as a promising avenue for future research.

Summary updated

Abstract

Compositional visual question answering (VQA) represents a challenging yet fundamental task that requires models to comprehend novel combinations of previously learned concepts. The current methods often overlook the disentanglement of underlying concepts and are restricted in terms of their ability to effectively capture the compositional variation mechanism. Moreover, the state-of-the-art techniques depend on additional clues for training, which is not feasible in real-world VQA scenarios. To address these issues, in this paper, we introduce a novel Disentanglement-based EquivAriant Learning (DEAL) framework for compositional VQA, which is guided exclusively by ground-truth answers. In DEAL, we employ causality-inspired interventions to disentangle concepts derived from visual and textual inputs within a re-encoding framework. Based on the principle of equivariance, we subsequently perform a compositional transformation on the inference input and impose the equivariant constraint on the output to augment the compositional reasoning capacity of the model. Comprehensive experiments conducted on the benchmark CLEVR-CoGenT and GQA-SGL datasets validate the superiority of our proposed DEAL approach over the existing state-of-the-art methods for compositional VQA tasks in both visual and linguistic generalization settings.
BibTeX
@article{du2025disentanglement,
  author = {Zhou Du and Zhaoquan Yuan and Xiao Wu and Changsheng Xu},
  title = {{Disentanglement-Based Equivariant Learning for Compositional VQA}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  doi = {10.1109/tmm.2025.3604897},
}
Figure