Experimentation is central to the machine learning process. During modeling, data scientists and machine learning engineers (MLEs) run a series of experiments to identify a robust predictive model. Typically, many models—possibly hundreds or even thousands—will be trained and evaluated in order to identify the techniques, architectures, learning algorithms, and parameters that work best for a particular problem.
Historically, the parameters that define each experiment—that experiment’s hyperparameters—are manually tracked by data scientists. This is done via any one of a variety of methods including using a lab notebook, document, spreadsheet, file and folder naming conventions, log files, etc. The experiment management features of modern ML platforms eliminate the burden and fallibility of manual tracking by programmatically logging the parameters and results of each test run.
With experiment parameters and results readily accessible, access to visual plots can be provided to allow data scientists to easily compare the performance of different model versions. These plots can be generated automatically as each experiment is logged and made available to data scientists via an experiment results dashboard.
Some degree of support for experiment management and tracking is included in most End-to-End ML platform offerings. In addition, a number of specialized products exist as well.