TL;DR
This paper introduces RS4PD, a recommender system that uses machine learning to automatically suggest the best process discovery algorithm for a given event log. By learning from past experiments, RS4PD helps users bridge the gap between complex process mining tools and their data, making process discovery more accessible and efficient.
What Problem Does This Paper Address?
Process mining aims to extract valuable insights from event logs, which are records of business process executions. A crucial step in process mining is "process discovery," where algorithms generate a process model from an event log. However, the field faces a significant challenge: there's no single "best" discovery algorithm. Different algorithms produce models with varying quality characteristics (e.g., fitness, precision, simplicity) and are based on different formalisms (like Petri nets or BPMN). This makes it difficult for users, especially those new to process mining, to select the most appropriate algorithm for their specific event log. Experienced users also face a tedious manual process of inspecting logs and experimenting with algorithms. This paper addresses this by proposing an automated solution to recommend the most suitable discovery algorithm.
Methods
The core of RS4PD is an Algorithm Recommender System (ARS) approach, adapted for process discovery. This system leverages a portfolio-based algorithm selection strategy. The process involves two main phases:
- Training: The system continuously runs experiments, executing various discovery algorithms on a diverse set of event logs. It collects performance and conformance metrics for each experiment. This data, along with extracted features from the event logs (e.g., trace length, number of distinct events, flow characteristics), is used to train prediction models. These models learn to associate log features with algorithm performance rankings for different quality measures.
- Recommending: When a user provides a new event log, RS4PD extracts its features. It then uses the trained prediction models to predict the performance ranking of algorithms for various quality measures. These predictions are combined into a final ranking, and the system recommends the top-k best-performing discovery techniques.
Key novelties include:
- Feature Engineering: Defining and extracting relevant features from event logs to characterize them for the prediction models.
- Machine Learning for Algorithm Selection: Employing machine learning to predict algorithm performance, moving beyond simple rule-based systems.
- Server-Client Architecture: A scalable architecture where a server handles the computationally intensive training and knowledge generation, while clients (e.g., ProM plugins) provide recommendations to users.
Datasets
The paper describes experiments conducted using:
- Training Data: 130 event logs (112 synthetic and 18 real-life) collected from various sources and uploaded into the system's repository.
- Testing Data: 13 event logs from the 3TU repository and the testing dataset from [19]. This includes 4 real-life logs from the Business Process Intelligence (BPI) workshop challenges of 2012 and 2013.
No new benchmarks were introduced; instead, existing collections of event logs were utilized.
Key Insights and Ideas
- Log Features are Key: The effectiveness of the recommender system hinges on the ability to characterize event logs using a set of relevant features. Even simple features can be surprisingly effective in predicting algorithm performance.
- No One-Size-Fits-All: The paper strongly reinforces the idea that different process discovery algorithms excel under different conditions, measured by various quality criteria (fitness, precision, simplicity, etc.). This justifies the need for a recommender system.
- Continuous Learning is Crucial: The server-client architecture allows for lifelong learning, where the system's knowledge base grows with each new experiment, leading to potentially better recommendations over time.
- User-Centric Recommendations: By allowing users to assign weights to different quality measures, the system can tailor recommendations to individual user priorities, making the output more actionable.
Key Results
The evaluation of RS4PD demonstrated promising accuracy:
- Global System Accuracy: The average accuracy of predicting the top-1 best-performing technique across all measures and log types was 0.854.
- Top-3 Accuracy: When considering the top-3 recommended techniques, the global system accuracy increased to 0.963.
- Measure-Specific Accuracy: Accuracy varied by measure category, ranging from 0.67 (for precision measures on real-life logs) to 1.0 (for fitness measures).
- Log Type Impact: The system showed slightly different performance on real-life versus synthetic logs, with some measures being more accurate on one type than the other.
Qualitatively, the system successfully integrates machine learning and information retrieval techniques to automate a complex decision-making process in process mining.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Parameter Setting: The current implementation uses default parameters for discovery algorithms. Future work could involve instantiating algorithms with different parameter settings or developing parameter optimization strategies.
- Feature Set: While simple features proved effective, there's room for improvement and extension of the feature set to better characterize event logs.
- Algorithm Portfolio: The system can easily incorporate new discovery and conformance checking techniques, expanding its recommendation capabilities.
- User Feedback: Integrating user feedback (e.g., usefulness of recommendations, user goals) into the training loop could further enhance the system's accuracy and usability.
- Scalability: The server-client architecture is designed for scalability, and further improvements in the training process and repository management are envisioned.
Summary updated