Generalization

Generalization refers to your model's ability to adapt properly to new, previously unseen data, drawn from the same distribution as the one used to create the model.

Generalization

• Goal: predict well on new data drawn from (hidden) true distribution.
• Problem: we don't see the truth.
• We only get to sample from it.
• Goal: predict well on new data drawn from (hidden) true distribution.
• Problem: we don't see the truth.
• We only get to sample from it.
• If model h fits our current sample well, how can we trust it will predict well on other new samples?
• Theoretically:
• Interesting field: generalization theory
• Based on ideas of measuring model simplicity / complexity
• Intuition: formalization of Ockham's Razor principle
• The less complex a model is, the more likely that a good empirical result is not just due to the peculiarities of our sample
• Empirically:
• Asking: will our model do well on a new sample of data?
• Evaluate: get a new sample of data-call it the test set
• Good performance on the test set is a useful indicator of good performance on the new data in general:
• If the test set is large enough
• If we don't cheat by using the test set over and over

Three basic assumptions in all of the above:

1. We draw examples independently and identically (i.i.d.) at random from the distribution
2. The distribution is stationary: It doesn't change over time
3. We always pull from the same distribution: Including training, validation, and test sets
[{ "type": "thumb-down", "id": "missingTheInformationINeed", "label":"Missing the information I need" },{ "type": "thumb-down", "id": "tooComplicatedTooManySteps", "label":"Too complicated / too many steps" },{ "type": "thumb-down", "id": "outOfDate", "label":"Out of date" },{ "type": "thumb-down", "id": "samplesCodeIssue", "label":"Samples / code issue" },{ "type": "thumb-down", "id": "otherDown", "label":"Other" }]
[{ "type": "thumb-up", "id": "easyToUnderstand", "label":"Easy to understand" },{ "type": "thumb-up", "id": "solvedMyProblem", "label":"Solved my problem" },{ "type": "thumb-up", "id": "otherUp", "label":"Other" }]