ASK-the-Expert: Active Learning Based Knowledge Discovery Using the Expert

Kamalika Das, Ilya Avrekh, Bryan L. Matthews, Manali Sharma, Nikunj C. OzaECML/PKDD 2017 · DOI

TL;DR

This paper introduces ASK-the-Expert, an interactive tool that streamlines knowledge discovery by enabling subject matter experts (SMEs) to efficiently label data within a single framework. It addresses the significant time and resource burden of manual data review, particularly for identifying operationally significant anomalies in large datasets, by integrating active learning to intelligently select instances for SME attention.

What Problem Does This Paper Address?

Discovering meaningful insights from massive datasets often requires human expertise, especially when identifying rare but critical events. In fields like aviation safety, unsupervised anomaly detection algorithms can flag thousands of potential issues, but only a tiny fraction are truly operationally significant. Manually reviewing all these anomalies is incredibly time-consuming and costly, as it demands the time of highly specialized subject matter experts (SMEs). While active learning promises to reduce this burden by intelligently querying SMEs for labels, its practical adoption is hindered by the lack of user-friendly tools that integrate the learning process with the review and labeling workflow. This paper tackles the challenge of bridging this gap by providing an interactive system that makes active learning accessible and efficient for SMEs.

Methods

The core of ASK-the-Expert is an interactive system composed of two main components: a coordinator and an annotator.

  • Coordinator: This component acts as the central hub. It receives a ranked list of anomalies from an unsupervised anomaly detection algorithm. It then communicates iteratively with an active learning "learner" to gather information about instances selected for annotation. The coordinator also handles the crucial task of resolving labeling conflicts if multiple SMEs are involved, either through majority voting or by escalating to an "investigator." A novel aspect is its ability to automate the creation of new "rationale features" by combining or modifying existing data features based on the SME's textual feedback.
  • Annotator: This is the graphical user interface (GUI) that SMEs interact with. It's designed to be installed on the SME's end. When launched, it checks for new data packets (instances to be labeled) from the cloud. It then displays these instances, ranked by importance, along with the features identified as anomalous. SMEs can delve deeper into each instance to review individual features, feature interactions, and even contextual information (like flight paths on maps). The annotator allows SMEs to provide labels (e.g., "operationally significant" or "not operationally significant") and free-text rationales.

The system is built using Python with PyQt for the GUI and leverages libraries like Matplotlib for data plotting and GmapPlot for flight path visualization.

Datasets

The paper discusses using data related to aviation safety, specifically flight operational data. However, due to the proprietary nature of these "data cubes" for normal and anomalous flights, the authors state that the database for their demonstration will be hosted on their laptop. They do not introduce a new benchmark dataset but rather apply their tool to a real-world aviation safety scenario.

Key Insights and Ideas

  • Bridging the Active Learning Gap: The most significant insight is the recognition that a lack of interactive tools is a major barrier to adopting active learning in practical, real-world scenarios, despite its potential for saving resources. ASK-the-Expert directly addresses this by providing a seamless workflow.
  • SME Feedback as Feature Engineering: The system's ability to automatically generate new rationale features from SME-provided textual feedback is a powerful idea. This moves beyond simple labeling to actively enriching the data representation based on expert knowledge, potentially improving classifier performance significantly.
  • Context is Crucial for Anomalies: For complex data like flight operations, simply presenting an anomaly isn't enough. The tool's ability to provide contextual information, such as plotting flight paths in relation to other traffic, is vital for SMEs to make informed labeling decisions.
  • Flexibility Beyond Active Learning: The authors highlight that the annotator component can be used as a standalone review and labeling tool, even without the active learning component, making it versatile for various data annotation tasks.

Key Results

The paper focuses on the system's design and functionality rather than presenting extensive quantitative results from a large-scale deployment. However, they reference a related work (Sharma et al., [2]) that used a similar active learning method with rationales to achieve a 70% reduction in SME labeling effort compared to manual review, while maintaining comparable accuracy. For their demo, they plan to showcase the tool in real-time, demonstrating how SMEs can download data, review examples, and provide labels, with their feedback feeding back into the learner for the next iteration.

Limitations and Future Work

  • Proprietary Data: The authors acknowledge that the specific aviation safety data used is proprietary, necessitating the use of a local database for their demonstration. This implies that broader adoption would require access to or adaptation for different data formats and sources.
  • Scalability and Deployment: While the software is built with Python and common libraries, the paper doesn't detail the scalability of the coordinator and annotator for extremely large numbers of users or massive datasets in a production environment.
  • Future Work: The authors explicitly state that the software will be open-sourced for adaptation into other applications. This suggests a strong interest in seeing the tool applied to diverse domains facing label scarcity. They also imply that further development could focus on enhancing the contextual visualization and the sophistication of the rationale feature generation.

Summary updated

BibTeX
@inproceedings{das2017ask,
  author = {Kamalika Das and Ilya Avrekh and Bryan L. Matthews and Manali Sharma and Nikunj C. Oza},
  title = {{ASK-the-Expert: Active Learning Based Knowledge Discovery Using the Expert}},
  booktitle = {ECML/PKDD},
  year = {2017},
  doi = {10.1007/978-3-319-71273-4\_38},
}
Figure