Abstract:
In many machine learning applications, the relationship being modelled may change over time, a phenomenon called concept drift. Most existing approaches to handling concept drift have assumed an artificially narrow specification of the problem. In this thesis we explore some new approaches to concept drift which introduce several new algorithms to help bridge academic concept drift research and real data science applications. As a motivating example for our investigation, we consider a medical clinic where a decision support system is helping clinicians triage patients referred by GPs. As the triage policy evolves, the decision support system should be able to detect that its model has become outdated, and signal to a human expert that it requires retraining. We first introduce the multiple drift detector (MDD) framework, in which a detector for several different types of concept drift is constructed out of a single “narrow” drift detector. This allows for a more complete and interpretable monitoring of concept drift. Changes in the accuracy, precision, recall, label distribution, or instance distribution can be detected, whereas typically only changes in accuracy are detected by existing drift detectors. We also present a graphical interface for MDD to assist understanding of how a data stream is evolving. Next, we introduce the calibrated drift detection method (CDDM), an algorithm which makes use of probabilistic predictions of models to detect increases in the reducible error, and not the irreducible error, of a model. Both of these are detected by conventional drift detectors, which can result in unnecessary and expensive model retraining. Next, we present Bayesian drift detection method (BDDM), an algorithm which computes exact posterior probability distributions over possible drift locations and over the error rate of the model. This allows decisions about whether to retrain a model to be made on a rational, expected utility basis. We also introduce Bayes with adaptive forgetfulness (BWAF), which is a heuristic approximation of BDDM. Finally, we experimentally validate our novel drift detection methods. We demonstrate the circumstances under which CDDM is useful. We also demonstrate that BWAF is competitive on most metrics on standard benchmarks, and as well as a synthetic medical triage data stream.