Holdout observations (train and test split)

The Meridian model specification contains a holdout_id argument (a boolean array of dimensions \(G \times T\)) that can be used to specify a holdout sample. The KPI data of the holdout observations is ignored during model training (for example, MCMC posterior sampling), and does not affect the model likelihood or posterior density. Media data for the holdout observations is still used for model training, because it affects the adstocked media values for subsequent time periods.

The primary use of the holdout sample is for calculating out-of-sample goodness of fit metrics, such as R-squared. This is useful for comparing different model specifications, such as prior distribution strengths, provided that each model being compared uses the same holdout sample. There is no guarantee that the model with the best out-of-sample model fit is the best model for causal inference, but generally a better fitting model is preferred. Model misspecifications that lead to poor model fit can also cause bias in causal inference.

We recommend using a holdout sample that is fairly balanced across geos and time periods. In other words, use a holdout sample that has approximately the same number of holdout observations for each geo and approximately the same number of holdout observations for each time period. If the holdout sample is imbalanced, this can result in too few training observations to estimate the geo effect \(\tau_g\) for certain geos, or the time effect \(\mu_t\) for certain time periods. By default, Meridian does not specify a holdout sample. You must specify the holdout sample and ensure that it has a reasonable degree of balance.

Avoid holding out large, contiguous-in-time, chunks of data, such as at the end of the MMM time window, to assess forecast error in the KPI. Meridian isn't designed for forecasting the KPI, especially if there is strong trend and seasonality in the KPI. Instead, Meridian estimates the causal media impact and uses the knot-based approach to modeling trend and seasonality. The knot-based approach needs data near the knot to estimate the knot effectively. If large, contiguous-in-time, chunks of data are held out, there is no data near the knots within the held out period. In this case, the knot's posterior distribution is driven by the prior, which can result in poor forecasting.

Additionally, Meridian can be used to estimate the impact of both historical and future media because it assumes that model parameters which determine media impact are consistent over time.