Creating Dynamic Checklists via Bayesian Case-Based Reasoning: Towards Decent Working Conditions for All

Eirik Lund Flogard, Ole Jakob Mengshoel, Kerstin BachIJCAI 2022 · OpenReview

TL;DR

This paper introduces Context-aware Bayesian Case-Based Reasoning (CBCBR), a novel machine learning approach that creates dynamic, adaptive checklists for labor inspections. Unlike static checklists, CBCBR's checklists evolve in real-time based on inspector input, leading to a significant increase in the detection of workplace violations and improved safety.

What Problem Does This Paper Address?

Occupational health and safety remain a critical global concern, with millions of deaths annually attributed to workplace hazards. Governmental agencies conduct labor inspections using checklists to ensure compliance with regulations and promote safe working environments, aligning with UN's Sustainable Development Goal 8. However, existing methods for creating these checklists, even those employing machine learning, result in static tools. These static checklists fail to adapt to the nuances of individual inspections or new information that emerges during the process. This inflexibility can lead to missed violations and reduced inspection efficiency. The paper addresses this gap by proposing a method for creating dynamic checklists that continuously learn and adapt during an inspection.

Methods

The core contribution is the Context-aware Bayesian Case-Based Reasoning (CBCBR) framework. CBCBR builds upon a previous method called Bayesian Case-Based Reasoning (BCBR), which constructs static checklists. The key novelty in CBCBR is the integration of a context-aware naive Bayesian inference (NBI) model. This model allows the checklist to dynamically adapt as an inspector answers its items.

Here's a breakdown of the CBCBR process:

  1. Initial Checklist Generation: Similar to BCBR, CBCBR uses an NBI model to estimate probabilities of non-compliance for various items. It then retrieves relevant past cases from a case base to construct an initial candidate checklist of K items.
  2. Dynamic Adaptation: As an inspector answers items on the checklist, this input is used to update the NBI model's posterior probability estimates. This updated model then informs recommendations for new items to be added to the checklist. The adaptation is driven by the idea that answers to existing items provide context that can predict the likelihood of non-compliance for other, unasked items.
  3. Case Base Augmentation: The case base, which stores past inspection data, is continuously updated with these new probability estimates, enabling more informed future recommendations.

The NBI model uses empirical distributions from the data to estimate probabilities, and a specific formulation (Equation 3) is introduced to enable online adaptation based on observed answers.

Datasets

The experiments were conducted using the Checklist data set previously introduced by Flogard et al. [2021]. This dataset contains 1,111,502 entries from 63,634 inspections, involving 369 unique checklists and 1,947 unique items. The data covers a wide range of industries, with checklist items typically related to specific inspection topics within those industries.

Key Insights and Ideas

  • Dynamic Checklists are More Effective: The fundamental insight is that checklists are not static tools. Their effectiveness can be significantly boosted by allowing them to adapt in real-time to the specific context of an inspection.
  • Answers Provide Rich Context: The answers provided by an inspector to initial checklist items are valuable signals that can predict the likelihood of non-compliance for other, unasked questions. This is the core principle behind CBCBR's dynamic adaptation.
  • Bayesian Inference for Real-time Adaptation: Using Bayesian inference, particularly the NBI model, allows for continuous updating of probability estimates, making real-time adaptation feasible.
  • Balancing Breadth and Depth: CBCBR's ability to recommend additional items allows inspectors to delve deeper into potentially high-risk areas, increasing the chances of finding violations without sacrificing the initial breadth of coverage.

Key Results

The experimental evaluation demonstrated that CBCBR significantly outperforms static checklist generation methods, including BCBR and various other machine learning baselines (NN, RF, NBI, DT, LR), as well as manually created expert checklists (ECL).

  • Higher Violation Detection: CBCBR-generated checklists found an average of 9.13 violations, compared to 7.19 for BCBR.
  • Improved Precision for Recommendations: The recommended items added by CBCBR showed a precision of 0.63, indicating that these additions were highly relevant.
  • Comparable Training Time: CBCBR's initial checklist generation time was comparable to BCBR (8.6 seconds), and updating the checklist with recommendations took only 4.5 seconds. While the cross-validation took over 20 days due to the nature of the evaluation, the individual update times suggest suitability for online use.
  • Overall Performance: CBCBR achieved the highest "Avg" score (a composite of accuracy, precision, and recall) among all tested methods.

Limitations and Future Work

The authors acknowledge several points and suggest future research directions:

  • Static Nature of Baselines: The primary limitation of previous work is the static nature of the generated checklists. CBCBR directly addresses this.
  • Simulation for Evaluation: The experiment relied on a simulator to generate checklist answers for predictive recommendations. While realistic, real-world testing is ongoing.
  • Future Directions:
    • Alternative Dynamic Adaptation Strategies: Exploring other ways to dynamically adapt checklists, such as building them incrementally from a single item.
  • Automated Checklist Item Generation: Investigating methods for automatically generating new checklist items, potentially building on Natural Language Processing techniques.
  • Other ML Methods: Exploring other machine learning models, such as Recurrent Neural Networks (RNNs), for dynamic checklist adaptation.
  • Broader Applications: Testing CBCBR in other domains beyond labor inspections, such as food safety, aviation, or medical surgeries.

Summary updated

Abstract

Every year there are 1.9 million deaths world-wide attributed to occupational health and safety risk factors. To address poor working conditions and fulfill UN's SDG 8, "protect labour rights and promote safe working environments for all workers", governmental agencies conduct labour inspections, using checklists to survey individual organisations for working environment violations. Recent research highlights the benefits of using machine learning for creating checklists. However, the current methods only create static checklists and do not adapt them to new information that surfaces during use. In contrast, we propose a new method called Context-aware Bayesian Case-Based Reasoning (CBCBR) that creates dynamic checklists. These checklists are continuously adapted as the inspections progress, based on how they are answered. Our evaluations show that CBCBR's dynamic checklists outperform static checklists created via the current state-of-the-art methods, increasing the expected number of working environment violations found in the labour inspections.
BibTeX
@inproceedings{flogard2022creating,
  author = {Eirik Lund Flogard and Ole Jakob Mengshoel and Kerstin Bach},
  title = {{Creating Dynamic Checklists via Bayesian Case-Based Reasoning: Towards Decent Working Conditions for All}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=EIF3RYiRpG},
}
Figure