In which of the following scenarios would a high accuracy value suggest
that the ML model is doing a good job?
A deadly, but curable, medical condition afflicts .01% of the
population. An ML model uses symptoms as features and predicts
this affliction with an accuracy of 99.99%.
Accuracy is a poor metric here. After all, even a "dumb" model
that always predicts "not sick" would still be 99.99% accurate.
Mistakenly predicting "not sick" for a person who actually is sick
could be deadly.
An expensive robotic chicken crosses a very busy road a
thousand times per day. An ML model evaluates traffic patterns and
predicts when this chicken can safely cross the street with an
accuracy of 99.99%.
A 99.99% accuracy value on a very busy road strongly suggests that
the ML model is far better than chance. In some settings, however,
the cost of making even a small number of mistakes is still too high.
99.99% accuracy means that the expensive chicken will need to be
replaced, on average, every 10 days. (The chicken might also cause
extensive damage to cars that it hits.)
In the game of roulette, a ball
is dropped on a spinning wheel and eventually lands in one of 38
slots. Using visual features (the spin of the ball, the position of
the wheel when the ball was dropped, the height of the ball over the
wheel), an ML model can predict the slot that the ball will land in
with an accuracy of 4%.
This ML model is making predictions far better than chance; a random
guess would be correct 1/38 of the time—yielding an accuracy of 2.6%.
Although the model's accuracy is "only" 4%, the benefits of success
far outweigh the disadvantages of failure.