Exercise 1: Start Clearly and Simply
Write what you'd like the machine learned model to do.
We want the machine learned model to:
| Tips for Success |
|---|
| At this point, the statement can be qualitative, but make sure this
captures your real goal, not an indirect goal.
If you are unsure, take a look at the previous lessons, and see what machine
learned models can possibly do. |
Exercise 2: Your Ideal Outcome
Adding your ML model to your system should produce a desirable outcome.
What is this outcome, independent of the model itself? Note that this outcome
may be quite different from how you assess the model and its quality.
Our ideal outcome is:
| Tips for Success |
|---|
| You don't need to limit yourself to metrics for which your product has
already been optimizing (those will be covered in the next exercise).
Instead, try to focus on the larger objective of your product or service.
|
Exercise 3: Your Success Metrics
Write down your metrics for success and failure with the ML system.
The failure metrics are important, that is, how will you know whether the ML
system has failed?
Both the success and failure metrics should be phrased independently of
evaluation metrics for the model.
For example, don't talk about precision, recall, or AUC; talk about the
anticipated outcomes, instead.
Frequently these metrics will be tied to the ideal outcome you specified above.
Our success metrics are:
Our ML model is deemed a failure if:
| Tips for Success |
|---|
- Are the metrics measurable?
- How will you measure them?
- When will you be able to measure them?
- How long will it take for you to know whether your new ML system is a
success or failure?
- Consider engineering and maintenance costs over the long-term.
- Failure may not only be caused simply by non achievement of a success
metric.
|
Exercise 4: Your Output
Write the output that you want your ML model to produce.
The output from our ML model will be:
The output is defined as one of the following:
- Unidimensional regression
- Multidimensional regression
- Binary classification
- Multiclass classification
- Generate text, image, audio, video, or some combination (multimodal)
| Tips for Success |
|---|
- The output must be quantifiable with a definition that a machine can
produce.
-
If you're using predictive ML, are you able to obtain example outputs to
use for training data?
- How and from what source?
- Your output examples may need to be engineered, such as turning watch time
into a percentile.
- If it is difficult to obtain example outputs to use for training,
you may need to revisit your responses to past exercises to reformulate your
problem and goals into ones that will allow you to train a model on your data.
|
Exercise 5: Using the Output
Write when your output must be obtained from the ML model, and how it is used in
your product.
The output from the ML model will be obtained when:
The outcome will be used for:
| Tips for Success |
|---|
Consider what data you'll need to make the prediction. For instance,
if you're using ML to predict travel time in a maps app, this might be
written as, "The output from the model will be obtained when the user
selects origin, destination, and mode of transportation."
|
Consider how you will use the predicted outcome in your product.
- Will it be presented immediately to the user in a UI?
- Will it be consumed by subsequent business logic?
- What latency requirements do you have?
Those requirements (serving requirement of the ML model) can impact what
information can be used to make predictions. For example:
- The latency of using data from remote services may make them
infeasible to use.
- If data sources lag in making new information available:
- Processed logs may be generated only once a day.
- Certain information is not known until it actually happens (such as
conversion events).
The Oracle Test: assume you always had the correct answer. How would you use
that in your product?
|
Exercise 6: Your Heuristics
Write how you would solve the problem if you didn't use ML. For example, what
heuristics you might use.
If we didn't use ML, we would:
| Tips for Success |
|---|
| Think about a scenario where you need to deliver the product
tomorrow, and you can only hardcode the business logic. What would you do?
|
Click the button below to either print or save your responses as a .pdf.
Print Page