Examining L₁ Regularization

This exercise contains a small, slightly noisy, training data set. In this kind of setting, overfitting is a real concern. Regularization might help, but which form of regularization?

This exercise consists of five related tasks. To simplify comparisons across the five tasks, run each task in a separate tab. Notice that the thicknesses of the lines connecting FEATURES and OUTPUT represent the relative weights of each feature.

Task	Regularization Type	Regularization Rate (lambda)
1	L2	0.1
2	L2	0.3
3	L1	0.1
4	L1	0.3
5	L1	experiment

Questions:

1. How does switching from L₂ to L₁ regularization influence the delta between test loss and training loss?
2. How does switching from L₂ to L₁ regularization influence the learned weights?
3. How does increasing the L₁ regularization rate (lambda) influence the learned weights?

(Answers appear just below the exercise.)

---

---

Click the plus icon for answers.

1. Switching from L₂ to L₁ regularization dramatically reduces the delta between test loss and training loss.
2. Switching from L₂ to L₁ regularization dampens all of the learned weights.
3. Increasing the L₁ regularization rate generally dampens the learned weights; however, if the regularization rate goes too high, the model can't converge and losses are very high.