Considerations in Selecting Drivers: The Law of Parsimony & Data Leakage

A long-standing principle applied to statistics and machine learning is the law of parsimony.  The idea is simply to build the simplest model with the least assumptions and variables but with the greatest explanatory power.  As it applies to machine learning, the temptation is to include all possible variables into the model no matter the relevance to the targeted outcome.  The thinking is more data = greater accuracy. While that may be the perception in the context of training a model, it often fails to generalize to data outside of what was used to train the model.  This can often result in overfitting the model and manifest itself in poor performance when applied to new data (out-of-sample). Robust models capture the quintessential inputs to best explain the outcome, and do so in the simplest way possible.