Experiment Management

Experiment Management

Experimentation is central to the machine learning process. During modeling, data scientists and machine learning engineers (MLEs) run a series of experiments to identify a robust predictive model. Typically, many models—possibly hundreds or even thousands—will be trained and evaluated in order to identify the techniques, architectures, learning algorithms, and parameters that work best for a particular problem.

Historically, the parameters that define each experiment—that experiment’s hyperparameters—are manually tracked by data scientists. This is done via any one of a variety of methods including using a lab notebook, document, spreadsheet, file and folder naming conventions, log files, etc. The experiment management features of modern ML platforms eliminate the burden and fallibility of manual tracking by programmatically logging the parameters and results of each test run.

With experiment parameters and results readily accessible, access to visual plots can be provided to allow data scientists to easily compare the performance of different model versions. These plots can be generated automatically as each experiment is logged and made available to data scientists via an experiment results dashboard.

Some degree of support for experiment management and tracking is included in most End-to-End ML platform offerings. In addition, a number of specialized products exist as well.

Build the best models
SAS Visual Data Mining and Machine Learning
Solve the most complex analytical problems with a single, integrated, collaborative solution
The MLOps Platform
An enterprise-grade platform for agile, reproducible, and scalable machine learning
The enterprise feature store
Modern MLOps focused on speed and simplicity
Weights & Biases
With a few lines of code, save everything you need to debug, compare and reproduce your models
AI and machine learning model management and operations for enterprise data science teams
Spell is DLOps
Open-source version control system for machine learning projects