Traditional computer vision relies heavily on the regular grid. Convolutional Neural Networks (CNNs) slide fixed-size kernels over pixels, assuming that the most important relationships are always between immediate neighbors in a Cartesian plane. However, many complex visual tasks—from diagnosing cancer in gigapixel pathology slides to recognizing faces in extreme poses—require understanding relationships that transcend the local grid.
A growing body of research proposes a more flexible alternative: representing images as graphs of patches. By treating discrete image regions as nodes and their relationships as edges, these methods allow models to reason about geometry, context, and non-local similarities that traditional convolutions often miss.
From Pixels to Patch Nodes
The first step in this shift is decomposing an image into meaningful units. While Soille et al. and Najman et al. explored representing images as graphs of pixels to apply mathematical morphology, modern deep learning approaches typically use image patches.
As Ciurte et al. and the authors argue, using patches instead of individual pixels provides inherent robustness to noise. In ultrasound imaging, for instance, individual pixels are plagued by speckle noise; by treating a small intensity patch as a single node, the graph representation captures local texture and context, effectively filtering out granular artifacts.
Once an image is divided into patches, a feature encoder (often a pre-trained CNN) transforms each patch into a high-dimensional vector. These vectors become the nodes of the graph.
Defining Connectivity and Context
The power of a graph representation lies in its edges. Unlike the rigid adjacency of a grid, graph edges can be tailored to the specific structural needs of a problem:
- Spatial Proximity: In digital pathology, the spatial arrangement of cells is critical. Raza et al. and the authors use Delaunay triangulation to connect neighboring tissue patches, ensuring the model learns the morphology of the tumor microenvironment. Similarly, Bontempo et al. and the authors use k-nearest neighbor (k-NN) graphs to preserve local context in large-scale slides that would otherwise be lost in traditional "bag-of-instances" approaches.
- Feature Similarity: Edges can also connect patches that look similar, regardless of their distance. Hancock et al. and Li et al. demonstrate that connecting patches based on non-local self-similarity allows for superior image fusion, preserving textures that CNNs might blur.
- Dynamic Relationships: In vehicle re-identification, Jiao et al. and the authors propose "dynamic erasure" to prune noisy background connections, forcing the graph to focus on discriminative features of the vehicle itself.
Why Graph Convolutions Work for Images
Once the graph is constructed, researchers apply Graph Neural Networks (GNNs) or Graph Convolutional Networks (GCNs) to process the data. This works because GNNs perform "message passing"—each patch node updates its own representation by aggregating information from its neighbors.
This mechanism is particularly effective for capturing geometric structure. Lee et al. and the authors adapted EdgeConv, a technique originally designed for 3D point clouds, to 2D monocular depth estimation. By treating patches as nodes, they bridged the gap between standard vision and 3D structural analysis, allowing the model to "see" the underlying geometry of a scene more clearly than a standard convolution could.
In social and environmental contexts, this relational reasoning allows for multi-view integration. Lee et al. and the authors used GCNs to link crowd-sourced street-level images. By modeling a neighborhood as a graph of viewpoints, they could predict livelihood indicators like poverty and health by capturing the contextual dependencies between different street corners.
Applications in Complex Data
The patch-graph approach is especially dominant in fields where data is irregular or multimodal:
- Digital Pathology: For cancer survival prediction, Zheng et al. and the authors use graph attention to fuse visual tissue patches with gene expression data. This allows the model to highlight "survival activation maps"—specific regions of the tissue graph that are most predictive of patient outcomes.
- Medical Diagnostics: Wang et al. and the authors developed GazeGNN, which integrates a radiologist's eye-gaze data directly into a patch graph of a chest X-ray. The graph structure allows the model to prioritize the anatomical regions the expert actually looked at, improving classification speed and accuracy.
- Image Restoration: Bai et al. and the authors treat image deblurring as a signal processing problem on a graph. By modeling patches as signals on weighted graphs, they can estimate blur kernels more robustly by analyzing the relationships between pixel intensities.
By moving away from the fixed grid and toward the flexible graph, these researchers have created a framework that can adapt to the inherent structure of the data, whether that structure is a biological tissue, a city street, or the geometric contours of a vehicle.
Key Foundational Work
On Morphological Hierarchical Representations for Image Processing and Spatial Data Clustering (2010) — contributed 1 concept elements to this direction.
Face Recognition Using Face Patch Networks (2013) — contributed 1 concept elements to this direction.