Executive Summary
Machine learning models often need updating to reflect new business rules or regulations, but waiting to collect new training data can be a major bottleneck. AIMEE (AI Model Explorer and Editor) is an interactive tool that lets users edit a black-box model's behavior directly using human-interpretable rules. By synthetically generating data that reflects these new rules, AIMEE retrains the model to adopt updated decision boundaries without needing fresh real-world data.
The Challenge of Concept Drift
In real-world applications like loan approvals or insurance claims, the rules of the game change frequently. A machine learning model trained on historical data might suddenly become obsolete due to a new company policy or government regulation.
Traditionally, updating the model's decision boundaries requires gathering a new dataset that reflects these changes—a process that is slow and sometimes impossible in the short term. The challenge lies in updating a model's decision boundaries before new training data is available, effectively bridging the gap between human domain knowledge and black-box model behavior.
The AIMEE Workflow
AIMEE is an end-to-end system that stitches together several techniques to allow bidirectional interaction with ML models.
- Interpretable Surrogates: The system first extracts the logic of a black-box model into a "global surrogate" made of Boolean decision rules (specifically, Disjunctive Normal Form, or ORs of ANDs). This gives users a readable summary of what the model is currently doing.
- Rule Editing & Data Pre-processing (FROTE): Users can write new rules to correct or update the model. AIMEE takes these rules and uses a technique called FROTE (Feedback rule-driven oversampling for editing models). Similar to SMOTE, it selects a base population from the old data and synthetically oversamples and relabels instances to force the dataset to reflect the user's new rules. The black-box model is then retrained on this augmented dataset.
- High-Dimensional Visualization: To show users the impact of their edits, AIMEE visualizes the dataset in 2D. It uses Gower’s distance (which handles both categorical and numeric data) combined with UMAP for dimensionality reduction. A Voronoi diagram is overlaid and color-coded by class to clearly show the old vs. new decision boundaries.
- Model Comparison: Finally, the system compares the surrogate rules of the original model against the newly trained model, visualizing the semantic similarity and overlap so users can verify that their feedback was accurately absorbed.
Key Insights
- Surrogates as an Interface, Not Just an Explanation: Typically, surrogate models are used purely for explainability (XAI)—peeking inside the black box. AIMEE flips this by using surrogates as a control panel, allowing users to inject domain knowledge back into the model.
- Synthetic Data for Concept Drift: You don't necessarily need to wait for real-world data to catch up to new business rules. By synthetically oversampling existing data according to expert-defined rules, you can simulate the new environment and preemptively update the model.
- Visualizing the Unvisualizable: Combining type-agnostic distance metrics (Gower's) with UMAP and Voronoi tessellations provides a highly intuitive way for non-ML experts to visually verify how high-dimensional decision boundaries shift after an update.
Demonstration Results
As a demonstration system, AIMEE focuses on architecture and user interface rather than quantitative benchmarks. The primary result is the successful end-to-end integration of the system, which demonstrates:
- The ability to extract and display DNF rules with their corresponding support metrics.
- A functional UI that allows real-time feedback rule building alongside dataset embedding visualizations.
- The successful generation of 2D boundary maps that allow users to toggle between and compare the original and updated decision boundaries.
Summary updated