Introduction: The Performance-Compatibility Challenge
AI systems are frequently updated to incorporate new data and improve overall accuracy. However, these updates often introduce "new" errors on tasks the system previously handled correctly. This phenomenon creates a performance-compatibility tradeoff: while the new model might be more accurate on average, its inconsistency with the user's established mental model can shatter trust.
Prior approaches attempted to mitigate this by penalizing newly introduced errors globally. However, because different users interact with different subsets of a system's capabilities, a global penalty is often a blunt instrument that unnecessarily sacrifices performance. This research proposes a personalized approach to model updates, ensuring that improvements do not come at the cost of breaking the specific functionalities a particular user relies upon.
Methodology: Personalized Weighting and Model Portfolios
The core innovation lies in personalizing the objective (loss) function used during the model update process. By adjusting the "weight" of training samples based on an individual user's history, the update can be tailored to preserve compatibility where it matters most for that user.
The proposed method categorizes training samples into four groups:
- The general dataset ($D$)
- The user's specific interaction history ($D_i$)
- Samples where the legacy model was correct ($D^c$)
- Samples where the legacy model was correct specifically within the user's history ($D^c_i$)
A weight vector $W = (w_0, w_1, w_2, w_3)$ is used to assign importance to these categories. Rather than searching for a single optimal weight, the system maintains a model portfolio consisting of nine models ($m_0$ through $m_8$), each representing a different binary combination of these weights. For instance, $m_0$ serves as a non-personalized baseline, while other models focus more heavily on user-specific history or legacy correctness. A selection algorithm then identifies the best-performing model from this portfolio for each user based on their unique interaction features.
Experimental Evaluation
The approach was evaluated using three real-world datasets from human-computer interaction contexts, each tracking approximately 100 users over time:
- ASSISTments: Interactions within an online tutoring platform.
- Galaxy Zoo (GZ): Crowdsourced classifications of galaxy images.
- MOOC: Student interaction data from Massive Open Online Courses.
Key Results
The personalized approach demonstrated statistically significant improvements in the performance-compatibility tradeoff across all three datasets. For some users, the personalized objective function yielded up to a 300% improvement in the tradeoff compared to non-personalized baselines.
Crucially, the performance of the personalized models was found to be statistically indistinguishable from an "oracle" model—a theoretical ideal that always selects the best possible model for a user. This suggests that the selection algorithm is highly effective at matching users with the appropriate personalization strategy from the portfolio.
Conclusion and Future Work
By moving from global to personalized compatibility constraints, AI updates can become more seamless and reliable for individual users. While the current implementation uses binary weights and a discrete portfolio, future research could explore continuous weighting schemes for even finer control. Additionally, addressing the computational overhead of maintaining multiple models and investigating the long-term effects of sequential updates remain important areas for further study.
Summary updated