Weakly-Supervised Structured Output Learning with Flexible and Latent Graphs Using High-Order Loss Functions

Gustavo Carneiro, Tingying Peng, Christine Bayer, Nassir Navab

Introduction

In oncology, quantifying tumor oxygenation is critical for evaluating therapeutic efficacy. Specifically, researchers focus on "microcirculatory supply units" (MCSUs)—tissue regions supported by individual microvessels. These units are classified into four states: Normoxia, Chronic Hypoxia, Acute Hypoxia, and Necrosis.

Accurately mapping these units presents significant hurdles. While expert pathologists can provide image-level counts, generating pixel-perfect annotations for thousands of units across diverse tissue samples is labor-intensive and impractical. Furthermore, the biological variability in the number and spatial arrangement of MCSUs requires models that can handle flexible output sizes rather than fixed-length vectors.

Methodology: Flexible Latent Graphs

The authors address these challenges by proposing two structured output models that leverage weak supervision through image-level counts. Central to both approaches is the concept of a latent graph, where nodes represent individual MCSUs.

Overview of the proposed FLSSVM and DCNN architectures for weakly-supervised MCSU classification.

Model A: Flexible Latent Structure SVM (FLSSVM)

The FLSSVM explicitly constructs a graph using Minimum Spanning Tree (MST) clustering. The graph structure is treated as a latent variable, inferred during the learning process. By utilizing a high-order loss function, the model optimizes for global class counts across the entire image, effectively regularizing local predictions without requiring pixel-level labels.

Model B: Deep Convolutional Neural Network (DCNN)

The second approach utilizes a 6-layer DCNN. To enable training with weak supervision, the authors implemented a custom loss function that integrates standard cross-entropy with a high-order term. A "softmax temperature" parameter is employed to make the discrete counting process differentiable, allowing for standard backpropagation.

Key Results and Performance

The models were evaluated on a dataset of 89 multimodal microscopy image pairs (Immunofluorescence and HE stainings). Both architectures demonstrated high accuracy in estimating MCSU distributions.

Correlation analysis between DCNN predictions and ground truth for MCSU counts and percentages.
  • Accuracy: The DCNN achieved a Pearson correlation ($r^2$) of 0.85 for raw counts and approximately 0.85 for the percentage distribution of oxygenation classes. The FLSSVM followed closely with an $r^2$ of 0.79.
  • Efficiency: The DCNN is significantly more efficient for real-time applications, processing an image in just 0.35 seconds, whereas the multi-stage FLSSVM requires approximately 30 seconds.
  • Biological Plausibility: Interestingly, while the DCNN was more numerically precise, the FLSSVM produced results that better aligned with biological priors. The explicit spatial constraints of the FLSSVM graph prevented unnatural "clumping" of predicted units, a phenomenon occasionally observed in the DCNN outputs.

Conclusion

This research demonstrates that high-order loss functions combined with flexible latent graphs can effectively bridge the gap between weak image-level supervision and complex structured output tasks in medical imaging. Future work aims to hybridize these methods, incorporating the spatial priors of graph-based models directly into deep learning architectures to achieve both high speed and biological realism.

Summary updated

Abstract

We introduce two new structured output models that use a latent graph, which is flexible in terms of the number of nodes and structure, where the training process minimises a high-order loss function using a weakly annotated training set. These models are developed in the context of microscopy imaging of malignant tumours, where the estimation of the number and proportion of classes of microcirculatory supply units (MCSU) is important in the assessment of the efficacy of common cancer treatments (an MCSU is a region of the tumour tissue supplied by a microvessel). The proposed methodologies take as input multimodal microscopy images of a tumour, and estimate the number and proportion of MCSU classes. This estimation is facilitated by the use of an underlying latent graph (not present in the manual annotations), where each MCSU is represented by a node in this graph, labelled with the MCSU class and image location. The training process uses the manual weak annotations available, consisting of the number of MCSU classes per training image, where the training objective is the minimisation of a high-order loss function based on the norm of the error between the manual and estimated annotations. One of the models proposed is based on a new flexible latent structure support vector machine (FLSSVM) and the other is based on a deep convolutional neural network (DCNN) model. Using a dataset of 89 weakly annotated pairs of multimodal images from eight tumours, we show that the quantitative results from DCNN are superior, but the qualitative results from FLSSVM are better and both display high correlation values regarding the number and proportion of MCSU classes compared to the manual annotations.
BibTeX
@inproceedings{carneiro2015weakly,
  author = {Gustavo Carneiro and Tingying Peng and Christine Bayer and Nassir Navab},
  title = {{Weakly-Supervised Structured Output Learning with Flexible and Latent Graphs Using High-Order Loss Functions}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  doi = {10.1109/iccv.2015.81},
}
Figure