Interactive Visual Study of Multiple Attributes Learning Model of X-Ray Scattering Images

Wei (Celia) Xu, Xinyi Huang, Suphanut Jamonnak, Ye Zhao, Boyu Wang, Minh Hoai, Kevin G. Yager, Wei Xu 0020IEEE TVCG 2021 · DOI

TL;DR

This paper introduces an interactive visualization system designed to help domain scientists understand how deep learning models classify X-ray scattering images based on multiple structural attributes. By visualizing data in actual label, feature, and prediction spaces, the system allows users to explore model performance, identify errors, and gain insights into attribute relationships, ultimately aiding in model refinement and dataset improvement.

What Problem Does This Paper Address?

X-ray scattering is a powerful technique for understanding material structures, but interpreting the resulting images, which often contain multiple overlapping attributes, is challenging. Deep learning models have shown promise in automating this classification, but their "black box" nature makes it difficult for domain scientists to trust, debug, or improve them. Existing visualization tools for deep learning are primarily designed for natural images and don't adequately address the unique challenges of scientific image classification with multiple, potentially correlated, attributes. This paper aims to bridge this gap by providing an interactive visual analytics system tailored for X-ray scattering image analysis.

Methods

The core of the system is an interactive visualization interface that projects X-ray scattering images into three key spaces:

  • Actual Attribute (ACT) Space: Represents the ground truth labels assigned by domain experts.
  • Feature (FEA) Space: Captures the high-level features extracted by the ResNet model's convolutional layers (2048 dimensions).
  • Prediction (PRD) Space: Shows the model's predicted attribute probabilities.

Images are visualized as points in these 2D embedded spaces using t-SNE (and optionally PCA). Users can interactively select groups of images (e.g., using lasso tools) and see how these selections are highlighted across all three spaces. This allows for direct comparison of how the model's predictions align with actual labels and how features are represented.

A novel "attribute flower" visualization is introduced to represent individual images, clearly depicting true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) for each attribute. The system also incorporates tools for analyzing attribute co-existence, including correlation matrices and conditional entropy visualizations, to understand how attribute relationships impact model performance.

Datasets

The system was trained and evaluated using an open X-ray scattering dataset [36] containing approximately 1,000 images. These images represent various materials like semiconductors, nanoparticles, and polymer films. The attributes are either labeled by domain experts or synthetically generated. The paper does not introduce a new benchmark dataset but utilizes an existing one for demonstrating the system's capabilities.

Key Insights and Ideas

  • Multi-space visualization is crucial: Simultaneously viewing images in ACT, FEA, and PRD spaces reveals discrepancies between ground truth, learned features, and model predictions, highlighting potential issues.
  • Attribute co-existence matters: The system's ability to visualize attribute relationships (e.g., correlation, conditional entropy) is vital for understanding how these interdependencies affect model accuracy, especially for scientific images where attributes are often correlated.
  • Attribute flowers offer intuitive error analysis: The attribute flower visualization provides an immediate, per-image breakdown of prediction correctness for all attributes, simplifying error identification.
  • Domain scientists can discover model flaws: Case studies demonstrate how the system helps identify issues like mislabeled data, confusing attributes (e.g., different types of beamstops), and limitations in the model's ability to distinguish subtle structural variations.

Key Results

The paper presents qualitative results through case studies demonstrating the system's utility. For instance:

  • Case 1: The system helped identify that the model struggled to distinguish between "Linear beamstop," "Wedge Beamstop," and "Circular Beamstop" attributes, likely due to their small size and similar visual characteristics, suggesting a need for more data or model refinement.
  • Case 2: The system highlighted that the model had difficulty with the "Many rings" attribute, suggesting that its visual representation might require a more refined labeling scheme.
  • Case 3: The system revealed errors in a pre-trained model's predictions for "BCC" and "FCC" attributes, indicating potential issues with data processing or insufficient training data for these specific attributes.

User studies showed high satisfaction ratings (above 8.0 on a 0-9 scale) for the visualization and interaction functions, with specific praise for the interlinked embedded views and the ease of exploring outliers and testing hypotheses.

Limitations and Future Work

  • Visual Clutter: Overlapping transparent dots in the embedded spaces can create misleading error rates. Interactive lenses or jittering tools are suggested to mitigate this.
  • Attribute Granularity: The current system relies on existing scientific attribute categories, which may not always be granular enough (e.g., "ring" might need further subdivision).
  • Data Labeling: Domain scientist labels can be prone to errors, suggesting the need for better labeling tools.
  • Model Depth: The system currently only visualizes features from the final convolutional layer. Future work could extend this to analyze feature maps from other layers and even individual neurons.
  • User Training: While generally intuitive, some aspects, like the group study panel, might require more training for users.
  • Integration with Training: The system could be integrated directly into the training process.

Summary updated

BibTeX
@article{xu2021interactive,
  author = {Wei (Celia) Xu and Xinyi Huang and Suphanut Jamonnak and Ye Zhao and Boyu Wang and Minh Hoai and Kevin G. Yager and Wei Xu},
  title = {{Interactive Visual Study of Multiple Attributes Learning Model of X-Ray Scattering Images}},
  journal = {IEEE TVCG},
  year = {2021},
  doi = {10.1109/tvcg.2020.3030384},
}
Figure