Graph Neural Networks in Computer Vision - Architectures, Datasets and Common Approaches

Maciej Krzywda, Szymon Lukasik, Amir H. Gandomi

TL;DR

This paper provides a comprehensive survey of Graph Neural Networks (GNNs) applied to computer vision tasks. It categorizes GNN architectures, highlights key datasets, and analyzes the relationships between GNN research in computer vision and other machine learning domains, offering a valuable roadmap for future research.

What Problem Does This Paper Address?

The field of computer vision is rapidly evolving, with an increasing number of applications leveraging the power of Graph Neural Networks (GNNs). However, the sheer volume of research and the diverse range of GNN architectures, datasets, and application areas can make it challenging for researchers to navigate this landscape. This paper aims to address this by consolidating and analyzing the current state of GNNs in computer vision, providing a structured overview of architectures, datasets, and the interconnections within the research community.

Methods

The authors conducted an extensive literature review, querying databases like Scopus with keywords such as "Computer Vision" and "Object Detection." They then categorized the identified papers based on the subjective areas of the Conference on Computer Vision and Pattern Recognition (CVPR). The review focuses on three main perspectives:

  • Architectures: The paper details various GNN architectures and their derivatives, including Graph Convolutional Networks (GCNs), Graph Recurrent Networks (GRNs), and Graph Attention Networks (GATs), explaining their core mechanisms and how they are adapted for computer vision tasks.
  • Datasets: A comprehensive list of commonly used datasets for GNN-based computer vision studies is presented, along with their fields of application and characteristics.
  • Relationships and Inspirations: A graph-based analysis is used to visualize the connections between research papers, identifying influential works and potential sources of inspiration from outside the computer vision domain.

Datasets

The paper lists several commonly used datasets for GNN-based computer vision studies, including:

  • NTU-RGBD: For action and behavior recognition.
  • CUHK03: For biometrics, face, gesture, and body pose.
  • Skeleton-Kinetics: For action and behavior recognition.
  • Visual Genome: Used across scene analysis, low-level and physics-based vision, and mesh generation/reconstruction.
  • ShapeNet: For mesh generation and reconstruction.
  • ShanghaiTech: For video analysis and understanding.
  • COCO: For recognition (object detection, categorization).
  • CONSeP: For medical, biological, and cell microscopy.
  • Colorectal Cancer (CRC): For medical, biological, and cell microscopy.
  • MS-Caleb-1M: For computational photography, image and video synthesis.
  • Jilin-1 and SkySat: For motion and tracking.

The paper does not introduce any new benchmarks but rather consolidates the usage of existing ones.

Key Insights and Ideas

  • GNNs are increasingly vital for computer vision: The paper clearly demonstrates a significant and growing interest in applying GNNs to a wide array of computer vision problems, moving beyond traditional grid-based approaches.
  • Architectural diversity is key: The survey highlights the evolution of GNN architectures, from foundational GCNs to more sophisticated GATs and spatio-temporal variants, each tailored to capture specific relational information in visual data.
  • Interdisciplinary inspiration is crucial: The analysis of citation networks reveals that influential GNN research in computer vision often draws inspiration from foundational GNN work in other domains like natural language processing and general machine learning, emphasizing the interconnectedness of AI research.
  • Skeleton-based action recognition and biometrics are leading application areas: These subfields show a high concentration of GNN research and appear to be particularly influential in driving GNN advancements in computer vision.

Key Results

The paper's primary contribution is a structured overview and analysis rather than quantitative results from new experiments. However, it identifies:

  • Key GNN Architectures: Detailed descriptions of GCNs, GRNs, GATs, and their specialized variants like ST-GCN, STGAT, and various graph attention mechanisms.
  • Prominent Datasets: A consolidated list of datasets frequently used for benchmarking GNN applications in computer vision.
  • Influential Papers: Through citation analysis, the paper identifies foundational works like Scarselli et al. [1] and Kipf et al. [3] as highly impactful, along with other key contributions in areas like GATs [5].
  • Research Trends: The analysis suggests that skeleton-based action recognition and biometrics (face, gesture, body pose) are particularly active and influential areas for GNN research in computer vision.

Limitations and Future Work

  • Limitations: The authors acknowledge that the domain of computer vision lacks a strict taxonomy, which influenced their categorization approach. They also note that while they covered many papers, the review is not exhaustive.
  • Future Work: The paper implicitly suggests future research directions by highlighting areas with significant GNN activity and by providing a clear overview of existing architectures and datasets. The authors anticipate that their study will be useful for researchers in computer vision and beyond, implying a desire for continued exploration and application of GNNs.

Summary updated

Abstract

Graph Neural Networks (GNNs) are a family of graph networks inspired by mechanisms existing between nodes on a graph. In recent years there has been an increased interest in GNN and their derivatives, i.e., Graph Attention Networks (GAT), Graph Convolutional Networks (GCN), and Graph Recurrent Networks (GRN). An increase in their usability in computer vision is also observed. The number of GNN applications in this field continues to expand; it includes video analysis and understanding, action and behavior recognition, computational photography, image and video synthesis from zero or few shots, and many more. This contribution aims to collect papers published about GNN-based approaches towards computer vision. They are described and summarized from three perspectives. Firstly, we investigate the architectures of Graph Neural Networks and their derivatives used in this area to provide accurate and explainable recommendations for the ensuing investigations. As for the other aspect, we also present datasets used in these works. Finally, using graph analysis, we also examine relations between GNN-based studies in computer vision and potential sources of inspiration identified outside of this field.
BibTeX
@inproceedings{krzywda2022graph,
  author = {Maciej Krzywda and Szymon Lukasik and Amir H. Gandomi},
  title = {{Graph Neural Networks in Computer Vision - Architectures, Datasets and Common Approaches}},
  booktitle = {IJCNN},
  year = {2022},
  url = {https://openreview.net/forum?id=kADxEDreYX},
}
Figure