DETECTING MACHINE LEARNING MODEL DEGRADATION BASED ON JOINT ANALYSIS OF DATA DRIFT AND PERFORMANCE METRICS

Abstract

Machine learning models operate in environments whose statistical properties may change after deployment; therefore, keeping the model parameters unchanged does not guarantee stable predictive performance. At the same time, the presence of input data drift alone does not provide evidence of model degradation, while direct performance evaluation is often complicated when ground truth is delayed or only partially available. The aim of this study is to develop a method for detecting machine learning model degradation that combines the analysis of changes in input data with predictive performance monitoring and accounts for the availability of ground truth when making monitoring decisions. The proposed method separately evaluates input feature drift, changes in the distribution of predictions, and deviations of performance metrics from their reference values. The Population Stability Index is used to determine the levels of input feature drift and change in the distribution of predictions. When a sufficient number of verified observations is available, the model state is determined from input feature drift and changes in predictive performance, which makes it possible to distinguish stable operation, drift without confirmed degradation, deterioration in predictive performance without detected drift, and the simultaneous occurrence of both, and to determine the corresponding response level. When the number of verified observations is insufficient, only a risk assessment based on changes in input features and in the distribution of predictions is produced. This assessment is refined once ground truth becomes available for a sufficient number of observations. The method was implemented in an information system for monitoring machine learning model performance and evaluated on classification and regression tasks under controlled scenarios of data change. The experimental results showed that pronounced input feature drift may occur without deterioration in predictive performance, whereas critical model degradation may arise without substantial changes in the monitored input distributions. The results also demonstrated that a risk assessment produced before the predictions are verified can be refined once ground truth becomes available. The proposed approach makes it possible to distinguish statistical changes in data from actual deterioration in predictive performance and to avoid an unjustified conclusion that the model is stable when the information required to evaluate its performance is unavailable.