TL;DR
This paper introduces VTG-Net, a novel framework that combines Convolutional Neural Networks (CNNs) with Graph Convolutional Networks (GCNs) to improve retinal artery/vein classification. By representing retinal vasculature as a graph that preserves topological information and fusing CNN features with graph-based topological features, VTG-Net achieves state-of-the-art accuracy, addressing limitations of previous methods.
What Problem Does This Paper Address?
Accurate classification of retinal arteries and veins (A/V) is crucial for diagnosing various eye diseases, such as diabetic retinopathy, and cardiovascular conditions. While traditional graph-based methods can capture vessel topology, they struggle to leverage the rich, hierarchical features extracted by modern deep learning models like CNNs. Conversely, current CNN-based approaches often fail to effectively incorporate the intricate topological relationships of the vascular network, leading to issues like misclassifying single vessel segments or failing to connect broken vessel segments. This paper aims to bridge this gap by developing a method that synergistically combines the feature extraction power of CNNs with the topological reasoning capabilities of GCNs.
Methods
The proposed VTG-Net framework consists of three main steps:
- CNN Feature Extraction: A standard CNN (specifically, a U-Net architecture was used for training) is employed to extract hierarchical features from the input fundus images. The output of the last convolutional layer is used as the representative CNN features.
- Graph Representation Generation: This is the core novelty. The segmented vessel mask (obtained from the CNN output) is used to define the graph's structure. Each pixel in the segmented vessel mask becomes a node in the graph. If two adjacent pixels are classified as vessels, an edge is created between their corresponding nodes, preserving the vessel's connectivity. Crucially, non-vessel (background) pixels are also included as isolated nodes in the graph. This is a key differentiator, as it allows the GCN to potentially correct errors made by the CNN in vessel segmentation or classification by providing context. Node features are constructed by concatenating the extracted CNN features with the original RGB image channels.
- Graph Classification and Fusion: A multi-layer GCN is trained on the generated graph representation to extract topological features. The GCN learns by propagating information across the graph, considering both node features and edge connectivity. Finally, the output from the CNN (pixel-wise classification) and the GCN (graph-based topological classification) are fused to produce the final A/V classification. The paper explores different fusion strategies, including agreement-based voting and weighted averaging of class probabilities.
Datasets
The VTG-Net was evaluated on two datasets:
- AV-DRIVE dataset: A publicly available dataset used for training and testing.
- Tongren dataset: An in-house dataset collected by the authors, also used for training and testing.
The paper notes that the sizes of these datasets are relatively small, which is identified as a limitation.
Key Insights and Ideas
- Isolated Background Nodes Matter: Including non-vessel pixels as isolated nodes in the graph representation is a critical design choice. The authors hypothesize that this allows the GCN to correct CNN errors by providing contextual information, especially for broken or disconnected vessel segments.
- Synergy of CNN and GCN: The paper demonstrates that combining deep hierarchical features from CNNs with topological information from GCNs leads to superior performance compared to using either method alone. The GCN can leverage CNN features to understand local vessel characteristics while simultaneously reasoning about global connectivity.
- Topology Preservation is Key: The graph representation explicitly encodes the vessel's structure, allowing the GCN to learn relationships that might be missed by standard CNN receptive fields, particularly in complex or noisy regions.
Key Results
- State-of-the-Art Performance: VTG-Net achieved a mean accuracy of 98.11% on the AV-DRIVE dataset, outperforming existing state-of-the-art methods by approximately 2%.
- Improved Accuracy and Sensitivity: The GCN component significantly improved classification accuracy and sensitivity compared to CNN-only approaches.
- Ablation Studies:
- Removing isolated background nodes led to a degradation in accuracy, highlighting their importance.
- CNN features were found to be critical for graph classification; accuracy dropped significantly when only RGB image data was used as node features.
- Dilation of segmented vessels for edge arrangement showed improvement for small dilation rates but degradation with larger rates.
Limitations and Future Work
- Dataset Size: The authors acknowledge that the datasets used are relatively small, limiting the thoroughness of the evaluation. They plan to conduct training and validation on larger datasets in future work.
- Fusion Strategy: While effective, the paper mentions exploring various fusion options and suggests further investigation into optimal fusion strategies.
- Generalizability: The performance on larger, more diverse datasets would be a valuable next step to confirm the generalizability of VTG-Net.
Summary updated