Introduction
Technical logbooks—the shorthand records kept by mechanics and engineers—present a significant challenge for Natural Language Processing (NLP). These datasets are typically riddled with domain-specific jargon, typos, and non-standard abbreviations. More critically, they exhibit extreme class imbalance. While common maintenance tasks like "oil changes" may have thousands of entries, rare but critical failures may only appear a handful of times. Standard machine learning models often become biased toward majority classes, failing to identify the high-stakes, rare events essential for effective predictive maintenance.
The study utilizes seven datasets from the MaintNet repository, an open-source resource for maintenance language. These cover aviation (maintenance, accidents, safety), automotive (maintenance, crashes, hazards), and facility maintenance domains. These datasets vary in scale from 617 to over 74,000 instances, with some containing up to 70 distinct event classes, making them a rigorous testing ground for classification strategies.
Methods: The Feedback Loop Strategy
To address these challenges, the authors adapt a Feedback Loop (FL) resampling strategy. Originally developed for wildlife identification in computer vision, this iterative process forces models to focus on the most difficult examples rather than being overwhelmed by common ones:
- Initial Training: The model is first trained on a small, randomly undersampled subset where every class has an equal number of instances.
- Inference & Error Calculation: The model then performs inference over the entire training set to identify where it struggles most.
- Resampling the "Worst": Instead of random selection, the algorithm specifically selects the instances where the model had the highest error for the next round of training.
- Iteration: This loop repeats for a fixed number of iterations, continually forcing the model to confront the examples it finds most confusing.
The researchers evaluated this strategy against four baselines: standard over-sampling, standard under-sampling, random down-sampling, and a "use everything" baseline. These were tested across four architectures: Deep Neural Networks (DNN), Long Short-Term Memory (LSTM), Convolutional Neural Networks (CNN), and BERT.
Results and Discussion
The Feedback Loop strategy was the clear winner. Across nearly every combination of model and dataset, the FL approach yielded the highest F1 scores (a metric balancing precision and recall), providing a robust, "plug-and-play" solution for handling imbalance.
Key Insights
- The "Hardest" Examples are the Best Teachers: By prioritizing instances the model gets wrong, the feedback loop prevents the model from becoming complacent with easy-to-predict majority classes.
- CNNs Punch Above Their Weight: Surprisingly, CNNs often matched or exceeded BERT's performance. The short, fragmented nature of logbook text (often 2–20 tokens) seems to play into the strengths of CNNs, which excel at picking up local "keyword" patterns without needing the complex global context BERT requires.
- Context Scarcity: Recurrent models like LSTMs struggled significantly. The authors suggest this is because logbook entries are too brief to provide the sequential context LSTMs leverage effectively.
For example, on the aviation maintenance dataset (Avi-Main), the CNN model using the Feedback Loop achieved an F1 score of 0.94, compared to 0.92 for the baseline. Statistical analysis using the Mann-Whitney U-test confirmed that these improvements were significant across all tested domains.
Limitations and Future Work
While effective, the study identified several areas for further research. BERT's performance was likely hampered because it has not been exposed to the specific technical dialects of mechanics; the development of a domain-specific "Maint-BERT" could bridge this gap. Additionally, data scarcity remains a ceiling for deep learning in smaller datasets like Auto-Main. Future work should also explore the applicability of these strategies to multilingual technical logbooks, where abbreviation and grammar conventions differ significantly from English standards.
Summary updated