Static vs. Dynamic Training

Broadly speaking, there are two ways to train a model:

  • A static model is trained offline. That is, we train the model exactly once and then use that trained model for a while.
  • A dynamic model is trained online. That is, data is continually entering the system and we're incorporating that data into the model through continuous updates.

Static vs. Dynamic Training

ML System Paradigms: Training

Static Model -- Trained Offline

ML System Paradigms: Training

Static Model -- Trained Offline

Dynamic Model -- Trained Online

ML System Paradigms: Training

Static Model -- Trained Offline

  • Easy to build and test -- use batch train & test, iterate until good.

Dynamic Model -- Trained Online

ML System Paradigms: Training

Static Model -- Trained Offline

  • Easy to build and test -- use batch train & test, iterate until good.
  • Still requires monitoring of inputs

Dynamic Model -- Trained Online

ML System Paradigms: Training

Static Model -- Trained Offline

  • Easy to build and test -- use batch train & test, iterate until good.
  • Still requires monitoring of inputs
  • Easy to let this grow stale

Dynamic Model -- Trained Online

ML System Paradigms: Training

Static Model -- Trained Offline

  • Easy to build and test -- use batch train & test, iterate until good.
  • Still requires monitoring of inputs
  • Easy to let this grow stale

Dynamic Model -- Trained Online

  • Continue to feed in training data over time, regularly sync out updated version.
  • Use progressive validation rather than batch training & test.

ML System Paradigms: Training

Static Model -- Trained Offline

  • Easy to build and test -- use batch train & test, iterate until good.
  • Still requires monitoring of inputs
  • Easy to let this grow stale

Dynamic Model -- Trained Online

  • Continue to feed in training data over time, regularly sync out updated version.
  • Use progressive validation rather than batch training & test
  • Needs monitoring, model rollback & data quarantine capabilities
  • Will adapt to changes, staleness issues avoided

Video Lecture Summary

Broadly speaking, the following points dominate the static vs. dynamic training decision:

  • Static models are easier to build and test.
  • Dynamic models adapt to changing data. The world is a highly changeable place. Sales predictions built from last year's data are unlikely to successfully predict next year's results.

If your data set truly isn't changing over time, choose static training because it is cheaper to create and maintain than dynamic training. However, many information sources really do change over time, even those with features that you think are as constant as, say, sea level. The moral: even with static training, you must still monitor your input data for change.

For example, consider a model trained to predict the probability that users will buy flowers. Because of time pressure, the model is trained only once using a dataset of flower buying behavior during July and August. The model is then shipped off to serve predictions in production, but is never updated. The model works fine for several months, but then makes terrible predictions around Valentine's Day because user behavior during that holiday period changes dramatically.

Send feedback about...

Machine Learning Crash Course