FDive: Learning Relevance Models using Pattern-based Similarity Measures

Frederik L. Dennig, Tom Polk, Zudi Lin, Tobias Schreck, Hanspeter Pfister, Michael Behrisch

Abstract

The detection of interesting patterns in large high-dimensional datasets is difficult because of their dimensionality and pattern complexity. Therefore, analysts require automated support for the extraction of relevant patterns. In this paper, we present FDive, a visual active learning system that helps to create visually explorable relevance models, assisted by learning a pattern-based similarity. We use a small set of user-provided labels to rank similarity measures, consisting of feature descriptor and distance function combinations, by their ability to distinguish relevant from irrelevant data. Based on the best-ranked similarity measure, the system calculates an interactive Self-Organizing Map-based relevance model, which classifies data according to the cluster affiliation. It also automatically prompts further relevance feedback to improve its accuracy. Uncertain areas, especially near the decision boundaries, are highlighted and can be refined by the user. We evaluate our approach by comparison to state-of-the-art feature selection techniques and demonstrate the usefulness of our approach by a case study classifying electron microscopy images of brain cells. The results show that FDive enhances both the quality and understanding of relevance models and can thus lead to new insights for brain research.

Bridging the Gap Between Human Intuition and High-Dimensional Data

In fields like neuroscience or computer vision, analysts often need to find "relevant" patterns—such as a specific type of cell synapse—within massive, high-dimensional datasets. Traditionally, this requires either a massive labeled training set for Deep Learning or a domain expert who knows exactly which mathematical features and distance metrics will best distinguish the data.

The "gap" is that experts often have the intuition to recognize a pattern but lack the technical expertise to pick the right mathematical "similarity measure" (the combination of a feature descriptor like a color histogram and a distance function like Euclidean distance). FDive tackles this by turning the selection of these measures into an automated, interactive process that learns from the user’s feedback in real-time.

The FDive Workflow

The FDive system operates in a four-step iterative loop designed to minimize the manual labeling burden on the analyst while maximizing model accuracy.

The FDive iterative workflow: (1) user provides relevance feedback, (2) the system suggests the best similarity measures, and (3) a hierarchical model is learned and steered.
  1. Relevance Feedback: The user labels a small subset of data as "relevant" or "irrelevant."
  2. Similarity Advisor: This is the core innovation. Instead of just selecting individual features, the system evaluates all possible pairs of Feature Descriptors (FDs) and Distance Functions. It ranks these pairs based on a quality metric called Inter-Group-Distance—essentially, how well a specific measure pushes the "relevant" and "irrelevant" clusters apart in mathematical space.
  3. Model Learning: The system uses the top-ranked similarity measure to train a Hierarchical Self-Organizing Map (SOM). Unlike a standard flat SOM, this creates a tree-like structure that partitions the data into increasingly fine-grained clusters.
  4. Steering & Refinement: The system highlights "uncertain" regions (cells in the SOM with mixed labels or low data density) and asks the user for more labels, specifically targeting the decision boundaries.

The system includes 24 different feature descriptors (covering color, texture, and shape) and various $L^p$ Minkowski distance norms to ensure a broad search space for the optimal model.

Key Insights and Ideas

Measures over Features

Most systems focus on "feature selection." FDive argues that the distance function is just as critical as the feature itself. A specific texture descriptor might fail with Euclidean distance but succeed with a Manhattan distance. By ranking combinations, FDive finds the most effective mathematical "lens" for the specific task.

Visualizing Uncertainty

By using a SOM, the system provides a "map" of the model's logic. When a user sees a yellow cell (indicating a mix of relevant and irrelevant items), they immediately know where the model is confused and where their labeling effort will be most impactful. This visual steering allows the user to act as a guide for the machine learning process.

The hierarchical Self-Organizing Map (SOM) interface allows users to explore clusters and identify regions of uncertainty for further labeling.

The "One-Shot" Advantage

The Similarity Advisor can often identify a high-performing measure with very few labels, outperforming complex feature-selection algorithms that require more data to converge. This makes it particularly effective in "low-label" scenarios common in specialized scientific research.

Evaluation and Results

The authors evaluated FDive using two distinct types of data: synapse detection in high-resolution Electron Microscopy (EM) images and sketches from the Google "Quick, Draw!" dataset.

  • Neuroscience Case Study: A domain expert used FDive to build a synapse detection model. In just nine iterations, the system converged on a specific combination (Haralick texture features + Euclidean distance) that successfully separated the complex EM images.
  • Quantitative Performance: When tested against state-of-the-art feature selection techniques (like ReliefF and Recursive Elimination SVMs), FDive’s Similarity Advisor achieved higher $F_1$ scores for complex shapes (banana, crayon, monkey) using the same amount of labeled data.

Limitations and Future Work

While powerful, the system faces challenges with "mental map disruption." When the system switches to a better similarity measure, the visual layout of the SOM changes completely, which can be jarring for the user. Future work aims to expand the library of distance functions (including Cosine and Canberra distances) and improve the computational scalability of preprocessing multiple feature descriptors for massive datasets.

Summary updated

Abstract

The detection of interesting patterns in large high-dimensional datasets is difficult because of their dimensionality and pattern complexity. Therefore, analysts require automated support for the extraction of relevant patterns. In this paper, we present FDive, a visual active learning system that helps to create visually explorable relevance models, assisted by learning a pattern-based similarity. We use a small set of user-provided labels to rank similarity measures, consisting of feature descriptor and distance function combinations, by their ability to distinguish relevant from irrelevant data. Based on the best-ranked similarity measure, the system calculates an interactive Self-Organizing Map-based relevance model, which classifies data according to the cluster affiliation. It also automatically prompts further relevance feedback to improve its accuracy. Uncertain areas, especially near the decision boundaries, are highlighted and can be refined by the user. We evaluate our approach by comparison to state-of-the-art feature selection techniques and demonstrate the usefulness of our approach by a case study classifying electron microscopy images of brain cells. The results show that FDive enhances both the quality and understanding of relevance models and can thus lead to new insights for brain research.
BibTeX
@inproceedings{dennig2019fdive,
  author = {Frederik L. Dennig and Tom Polk and Zudi Lin and Tobias Schreck and Hanspeter Pfister and Michael Behrisch},
  title = {{FDive: Learning Relevance Models using Pattern-based Similarity Measures}},
  booktitle = {VAST},
  year = {2019},
  url = {https://openreview.net/forum?id=TP2AKBCBo-D},
}

Loading related content...

Figure