Regularization for Sparsity

Stay organized with collections Save and categorize content based on your preferences.

This module focuses on the special requirements for models learned on feature vectors that have many dimensions.

Regularization for Sparsity

  • Caveat: Sparse feature crosses may significantly increase feature space
  • Possible issues:
    • Model size (RAM) may become huge
    • "Noise" coefficients (causes overfitting)
  • Would like to penalize L0 norm of weights
    • Non-convex optimization; NP-hard
  • Would like to penalize L0 norm of weights
    • Non-convex optimization; NP-hard
  • Relax to L1 regularization:
    • Penalize sum of abs(weights)
    • Convex problem
    • Encourage sparsity unlike L2