Static vs. Dynamic Inference

You can choose either of the following inference strategies:

  • offline inference, meaning that you make all possible predictions in a batch, using a MapReduce or something similar. You then write the predictions to an SSTable or Bigtable, and then feed these to a cache/lookup table.
  • online inference, meaning that you predict on demand, using a server.

Learn more about static vs. dynamic inference in the following video (2 min).

Static vs. Dynamic Inference

ML System Paradigms: Inference

Offline Inference

  • Make all possible predictions in a batch, using a mapreduce or similar.
  • Write to a table, then feed these to a cache/lookup table.

Online Inference

  • Predict on demand, using a server.

ML System Paradigms: Inference

Offline Inference

  • Make all possible predictions in a batch, using a mapreduce or similar.
  • Write to a table, then feed these to a cache/lookup table.
  • Upside: don't need to worry much about cost of inference.
  • Upside: can likely use batch quota.
  • Upside: can do post-verification of predictions on data before pushing.

ML System Paradigms: Inference

Offline Inference

  • Make all possible predictions in a batch, using a mapreduce or similar.
  • Write to a table, then feed these to a cache/lookup table.
  • Upside: don't need to worry much about cost of inference.
  • Upside: can likely use batch quota.
  • Upside: can do post-verification on predictions on data before pushing.
  • Downside: can only predict things we know about -- bad for long tail.
  • Downside: update latency likely measured in hours or days.

ML System Paradigms: Inference

Online Inference

  • Predict on demand, using a server.
  • Upside: can predict any new item as it comes in -- great for long tail.

ML System Paradigms: Inference

Online Inference

  • Predict on demand, using a server.
  • Upside: can predict any new item as it comes in -- great for long tail.
  • Downside: compute intensive, latency sensitive -- may limit model complexity.
  • Downside: monitoring needs are more intensive.

Video Lecture Summary

Here are the pros and cons of offline inference:

  • Pro: Don’t need to worry much about cost of inference.
  • Pro: Can likely use batch quota or some giant MapReduce.
  • Pro: Can do post-verification of predictions before pushing.
  • Con: Can only predict things we know about — bad for long tail.
  • Con: Update latency is likely measured in hours or days.

Here are the pros and cons of online inference:

  • Pro: Can make a prediction on any new item as it comes in — great for long tail.
  • Con: Compute intensive, latency sensitive—may limit model complexity.
  • Con: Monitoring needs are more intensive.

Send feedback about...

Machine Learning Crash Course