Library

When Good Models Go Bad: The damage caused by wayward models and how to prevent it

Delivering a bad model into production/serving is deceptively easy to do and can create significant and difficult-to-mitigate damage. Problems with models range from simple issues like incompatibility with the serving system to more subtle quality regressions. Using a hand analysis of approximately 100 incidents tracked over 10 years, we look carefully at cases where these models reached, or almost reached the serving system. We identify common causes and manifestations of these failures and provide some ideas for how to measure the potential damage of various failures. Most importantly, we propose a set of simple (and some more sophisticated) techniques for detecting the problems before they cause damage.