AI Solution’s maintenance is different from traditional software
IDC predicts that up to 88 percent of all AI and ML projects will fail during the test phase[1]. Major reason is that AI solutions are difficult to maintain. In this post I will highlight how maintenance of AI solution is different and why MLOps are important.
Some business executives and even engineers think that when an AI solution is deployed, you’re done. But most of the time you may only be halfway to the goal. Substantial work lies ahead in monitoring, maintaining and optimizing the system. I will list down some of the maintenance challenges below:
Data drift:
Data drift is one of the top reasons model accuracy degrades over time. Data drift is the change in model input data that leads to model performance degradation. The model was trained on a certain distribution of data, but this distribution changes over time. Causes of data drift include:
- Upstream process changes, such as Customer profile and behavioral data model is updated that changes the incoming data into AI model for customer segmentation or personalized marketing
- Data quality issues, such as a broken sensor always reading 0 or data from some form get corrupted because of some bug that will send bad data into model
- Natural drift in the data, such as pre-covid vs post-covid changes in user behavior
For example, a model may have learned to estimate demand for a mass transit system from historical pre-covid data, but covid caused unprecedented changes to riders patterns, so the model’s accuracy will degrade. AI model will need to be retrained to reflect new realities.
Concept drift.
Concept drift is a major consideration for ensuring the long-term accuracy of AI solutions. Concept drift can be understood as changes in the relationship between the input and target output. The properties of the target variables may evolve and change over time. The model was trained to learn an x->y mapping, but the statistical relationship between x and y changes, so the same input x now demands a different prediction y.
Concept drift can be also be caused by redefinition of y variable. For instance, a model detects construction workers who wander into a dangerous area without a hard hat for more than 5 seconds. But safety requirements change, and now it must flag hatless workers who enter the area for more than 3 seconds. This will require retraining of the model.
This is an excerpt of the article, read full article here
