**Generalization** refers to your model's ability to adapt properly
to new, previously unseen data, drawn from the same distribution as the
one used to create the model.

# Generalization

## The Big Picture

- Goal: predict well on new data drawn from (hidden) true distribution.
- Problem: we don't see the truth.
- We only get to sample from it.

## The Big Picture

- Goal: predict well on new data drawn from (hidden) true distribution.
- Problem: we don't see the truth.
- We only get to sample from it.
- If model h fits our current sample well, how can we trust it will predict well on other new samples?

## How Do We Know If Our Model Is Good?

- Theoretically:
- Interesting field: generalization theory
- Based on ideas of measuring model simplicity / complexity
- Intuition: formalization of Occam's Razor principle
- The less complex a model is, the more likely that a good empirical result is not just due to the peculiarities of our sample

## How Do We Know If Our Model Is Good?

- Empirically:
- Asking: will our model do well on a new sample of data?
- Evaluate: get a new sample of data-call it the test set
- Good performance on the test set is a useful indicator of good performance on the new data in general:
- If the test set is large enough
- If we don't cheat by using the test set over and over

## The ML Fine Print

Three basic assumptions in all of the above:

- We draw examples
**independently and identically (i.i.d.)**at random from the distribution - The distribution is
**stationary**: It doesn't change over time - We always pull from the
**same distribution**: Including training, validation, and test sets