Beyond Labels: Knowledge Elicitation using Deep Metric Learning and Psychometric Testing

Lu YinIJCAI 2020 · OpenReview

Abstract

Knowledge present in a domain is well expressed as relationships between corresponding concepts. For example, in zoology, animal species form complex hierarchies; in genomics, the different (parts of) molecules are organized in groups and subgroups based on their functions; plants, molecules, and astronomical objects all form complex taxonomies. Nevertheless, when applying supervised machine learning (ML) in such domains, we commonly reduce the complex and rich knowledge to a fixed set of labels. This oversimplifies and limits the potential impact that the ML solution can deliver. The main reason for such a reductionist approach is the difficulty in eliciting the domain knowledge from the experts. Developing a label structure with sufficient fidelity and providing comprehensive multi-label annotation can be exceedingly labor-intensive in many real-world applications. Here, we provide a method for efficient hierarchical knowledge elicitation (HKE) from experts working with high-dimensional data such as images or videos. Our method is based on psychometric testing and active deep metric learning. The developed models embed the high-dimensional data in a metric space where distances are semantically meaningful, and the data can be organized in a hierarchical structure.

Introduction

In the world of supervised machine learning, experts are often forced to squeeze their vast, nuanced knowledge into a narrow pipe: the flat label. Whether it's "cat vs. dog" or "severity level 1 vs. 4," these discrete categories often fail to capture the rich, hierarchical relationships that exist in the real world. At IJCAI-20, Lu Yin presented a framework to bridge this gap, moving "beyond labels" to elicit complex domain knowledge through the lens of psychometric testing and deep metric learning.

The Challenge of Reductionist Labeling

Most machine learning models are "reductionist." In fields like medicine or zoology, experts understand data through complex taxonomies. However, standard supervised learning requires "flat" labels, which forces an expert to project their multi-dimensional knowledge onto a single axis. This creates two major issues:

  1. Information Loss: The model misses out on the "depth" of knowledge, limiting its performance and interpretability.
  2. Scalability: Manually annotating every data point with multiple hierarchical tags is prohibitively labor-intensive and prone to inconsistency.

The goal of Hierarchical Knowledge Elicitation (HKE) is to extract this "latent perception"—the subconscious rules an expert uses to distinguish objects—in a way that is efficient, scalable, and doesn't require pre-defined label structures.

The HKE Framework

The proposed HKE framework combines three distinct fields to automate the discovery of taxonomies:

The HKE workflow: from psychometric triplet tests to hierarchical clustering that matches human perception.
  • Psychometric Testing (3AFC): Instead of asking an expert "What is this?", the system uses a Three-Alternative-Forced-Choice (3AFC) test. The expert is shown three objects and asked: "Which one is the most different?" This is much easier for humans to answer consistently than assigning abstract labels.
  • Active Learning: To avoid asking every possible combination of triplets, the system uses a Bayesian-based active learning method to select triplets with high uncertainty and high utility.
  • Deep Metric Learning: Responses are fed into a Deep Neural Network using a dual-triplet loss with adaptive margin. This maps high-dimensional data into a lower-dimensional embedding space where distances represent semantic similarity.
  • Clustering: Once embedded in this "perceptual space," standard clustering techniques reveal the hierarchical structure.

Experimental Results: Capturing Human Perception

The framework was evaluated using a synthetic shape dataset varying in shape and color. This controlled environment allowed researchers to verify if the model could successfully recover known human cognitive biases, such as the "shape bias"—the tendency for humans to find shapes more fundamental for discrimination than colors.

The elicited hierarchy for the synthetic shape dataset, showing primary grouping by shape and secondary grouping by color.

As shown in the results, the HKE framework successfully uncovered the hierarchical structure. The model placed shape-based differences higher in the hierarchy than color-based differences, mirroring human responses. The active learning component significantly reduced the number of queries needed, while the dual-triplet loss proved effective at creating a clear, separable hierarchical structure in the embedding space.

Conclusion and Future Work

By focusing on "elicitation" (extracting relationships) over "labeling" (assigning names), the HKE model learns the structure of a domain without pre-defined categories. This reduces the "black-box" nature of AI by organizing data into hierarchies that match human expert intuition. Future work will explore handling "crowd" knowledge from multiple experts and applying the method to complex stimuli like voice recordings to elicit hierarchies of human emotions.

Summary updated

Abstract

Knowledge present in a domain is well expressed as relationships between corresponding concepts. For example, in zoology, animal species form complex hierarchies; in genomics, the different (parts of) molecules are organized in groups and subgroups based on their functions; plants, molecules, and astronomical objects all form complex taxonomies. Nevertheless, when applying supervised machine learning (ML) in such domains, we commonly reduce the complex and rich knowledge to a fixed set of labels. This oversimplifies and limits the potential impact that the ML solution can deliver. The main reason for such a reductionist approach is the difficulty in eliciting the domain knowledge from the experts. Developing a label structure with sufficient fidelity and providing comprehensive multi-label annotation can be exceedingly labor-intensive in many real-world applications. Here, we provide a method for efficient hierarchical knowledge elicitation (HKE) from experts working with high-dimensional data such as images or videos. Our method is based on psychometric testing and active deep metric learning. The developed models embed the high-dimensional data in a metric space where distances are semantically meaningful, and the data can be organized in a hierarchical structure.
BibTeX
@inproceedings{yin2020beyond,
  author = {Lu Yin},
  title = {{Beyond Labels: Knowledge Elicitation using Deep Metric Learning and Psychometric Testing}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=twtCAMATmF},
}
Figure