Feature Crosses

A feature cross is a synthetic feature formed by multiplying (crossing) two or more features. Crossing combinations of features can provide predictive abilities beyond what those features can provide individually.

Feature Crosses

  • Feature crosses is the name of this approach
  • Define templates of the form [A x B]
  • Can be complex: [A x B x C x D x E]
  • When A and B represent boolean features, such as bins, the resulting crosses can be extremely sparse

  • Housing market price predictor:

    [latitude X num_bedrooms]

  • Housing market price predictor:

    [latitude X num_bedrooms]

  • Tic-Tac-Toe predictor:

    [pos1 x pos2 x ... x pos9]

  • Linear learners use linear models
  • Such learners scale well to massive data e.g., Vowpal Wabbit, sofia-ml
  • But without feature crosses, the expressivity of these models would be limited
  • Using feature crosses + massive data is one efficient strategy for learning highly complex models
    • Foreshadowing: neural nets provide another
Previous
Cleaning Data
Next
Encoding Nonlinearity