An Adaptive Multi-Agent Framework for Semantic-Aware Process Mining

Xiaohan Su, Bin Liang, Zhidong Li, Yifei Dong, Justin Wang, Fang ChenComput 2025 · DOI

TL;DR

This paper introduces an adaptive multi-agent framework that uses Large Language Models (LLMs) to understand and analyze business processes by combining event logs with textual rules. It significantly improves semantic-aware process mining tasks like anomaly detection and next-activity prediction by dynamically learning from user feedback and cross-task information, making it robust to evolving process dynamics.

What Problem Does This Paper Address?

Traditional process mining relies heavily on structured event logs, often missing crucial contextual information found in textual documents like policies, case notes, and domain manuals. This gap limits the ability to accurately discover, check, and improve real-world processes, especially when decisions depend on semantic understanding rather than just event sequences. Furthermore, existing methods struggle to adapt to "process drift" – changes in rules or policies over time – and often require extensive offline fine-tuning. This paper tackles the challenge of building a process mining system that can effectively leverage textual knowledge, adapt to changes in real-time, and improve its performance based on user feedback without constant retraining.

Methods

The core of the proposed solution is an Adaptive Multi-Agent Framework that integrates Retrieval-Augmented Generation (RAG) with a multi-agent system. Key components include:

  • Unified Knowledge Base: A shared knowledge base stores information relevant to multiple process mining tasks (semantic next-activity prediction (S_NAP), trace-level semantic anomaly detection (T_SAD), and activity-level semantic anomaly detection (A_SAD)). This allows for cross-task corroboration, where insights from one task can inform another, improving coverage and consistency.
  • Online Feedback Calibration: User feedback (correctness/usefulness ratings) is used to update "index priors." These priors dynamically bias the retrieval of relevant information for future queries, enabling serve-time adaptivity without heavy fine-tuning.
  • Multi-Agent Architecture: Different agents are assigned to specific tasks (S_NAP, T_SAD, A_SAD), each leveraging the RAG module for context.
  • Stability Controls: The framework incorporates mechanisms like consistency-aware scoring, confidence gating (which can trigger query rewriting), and task-level trust regions to ensure stable performance and prevent cascading errors, especially under dynamic conditions.
  • Sequential Selection: An online selection rule for choosing retrieval parameters balances exploration (trying new information) and exploitation (using known good information).

Datasets

The experiments utilized data derived from the SAP-SAM (SAP Signavio Academic Models) collection. This collection consists of BPMN process models. For evaluation, the authors:

  • Generated allowed execution sequences from sound workflow nets.
  • Converted traces into PDF documents to form the RAG knowledge base.
  • Strictly separated the test set from the knowledge base at the process-model level to prevent data leakage.

The paper provides a data distribution table showing the total dataset size, the portion used for the RAG document base, and the test set size for each of the three tasks (T_SAD, A_SAD, S_NAP).

Key Insights and Ideas

  • Feedback is King (for Adaptation): The most significant driver of next-query improvements and better initial rankings is the "feedback-to-index prior" mechanism. This highlights that directly incorporating user feedback into the retrieval process is crucial for adaptation.
  • Cross-Task Synergy is Powerful: Leveraging a single shared knowledge base for multiple tasks (cross-task corroboration) significantly boosts performance, especially for S_NAP, demonstrating that semantic overlaps between tasks can be exploited.
  • Task Complexity Matters: The effectiveness of LLMs and the framework varies with task complexity. S_NAP, requiring deeper semantic reasoning, benefits more from larger models and the adaptive framework, while anomaly detection tasks show more modest gains, suggesting that simpler tasks might be less sensitive to model size but still benefit from adaptive retrieval.
  • Online Adaptation Outperforms Static Baselines: The framework's ability to adapt online to semantic drifts in processes, as shown in the streaming experiment, is a critical advantage over static systems that fail to recover after concept drift.

Key Results

The proposed framework consistently outperformed baselines, including in-context learning (ICL), retrieval-only, and correction-centric methods.

  • Overall Improvements: The framework achieved significant macro-F1 score improvements compared to ICL: 44.0% for S_NAP, 15.6% for T_SAD, and 7.1% for A_SAD.
  • Model Performance: GPT-40, a larger model, achieved the highest F1 score (0.81) on S_NAP. However, smaller models like Llama also showed substantial improvements with the framework, particularly on S_NAP.
  • Ablation Studies:
    • Removing index priors caused the steepest degradation.
  • Cross-task corroboration provided consistent gains across tasks.
  • Confidence gating enhanced robustness to difficult or noisy inputs.

Limitations and Future Work

  • LLM Size and Resource Requirements: Optimal performance may require advanced LLMs with larger parameter counts, which can be challenging for resource-limited environments.
  • Context Quality Dependency: The framework's performance is dependent on the RAG module's ability to generate accurate context. Imprecise retrieval can hinder agent decision-making.
  • Future Directions:
    • Optimizing resource usage through model distillation or lighter alternatives.
  • Improving context quality using advanced retrieval techniques or knowledge graphs.
  • Large-scale real-world validations to assess scalability and applicability.

Summary updated

BibTeX
@article{su2025adaptive,
  author = {Xiaohan Su and Bin Liang and Zhidong Li and Yifei Dong and Justin Wang and Fang Chen},
  title = {{An Adaptive Multi-Agent Framework for Semantic-Aware Process Mining}},
  journal = {Comput.},
  year = {2025},
  doi = {10.3390/computers14110481},
}
Figure