Meridian requires passing distributions for ROI calibration. Although setting custom priors using results from previous experiments is a sound approach, there are many nuances to consider before proceeding. For example:
The timing of the experiment in relation to the MMM time window: If the experiment was conducted either before or after the MMM time window, the results might not be directly applicable.
The duration of the experiment: An experiment of short duration might not effectively capture the long-term effects of the marketing effectiveness.
The complexity of the experiment: If the experiment involved a mixture of channels, the results might not provide clear insights into the performance of individual channels.
Estimand differences: The estimands used in experiments can differ from those used in the MMM. For example, the MMM counterfactual is zero spend, whereas some experiments might have a different counterfactual, such as reduced spend.
Population differences: The population targeted in the experiment might not be the same as the population considered in the MMM.
We recommend setting the custom priors based on your belief in the effectiveness of a channel. A prior belief can be informed by many things, including experiments or other reliable analyses. Use the strength in that prior belief to inform the standard deviation of the prior:
If you have a strong belief in the effectiveness of a channel, you can apply an adjustment factor to the standard deviation of the prior to reflect your confidence. For example, suppose you have conducted several experiments for a particular channel and all the experiments yielded similar ROI point estimates, or you have historical data from previous MMM analyses that support the effectiveness of this channel. In this case, you could set a smaller standard deviation for the prior so that the distribution won't vary widely. This tighter distribution indicates your strong confidence in the experimental results.
Conversely, the experiment might not necessarily translate to the MMM, considering some of the nuances listed earlier. In this case, you might choose to apply an adjustment factor to standard deviation of the prior distribution. For example, you could set a larger standard deviation for the prior, depending on your level of skepticism.
You should consider using the roi_calibration_period argument in
ModelSpec. For more information, see
Set the ROI calibration period.
Define a lognormal prior from intuition or experiments
The lognormal distribution is a common distribution to use for an ROI prior.
Meridian provides two helper functions to construct lognormal distributions
from experiment results.
prior_distribution.lognormal_dist_from_mean_std
constructs a lognormal distribution with a mean and standard deviation
provided by the user. For example, the chosen mean and standard deviation could
be informed by the point estimate and standard errors of experiments,
respectively.
prior_distribution.lognormal_dist_from_range
constructs a lognormal distribution so that a specified probability mass falls
within a range. For example, the range could be informed by a 95% confidence
interval from a previous experiment. When using either function to define a
prior from experiments, keep the considerations discussed on this page in mind.
To construct a LogNormal distribution from a mean and standard deviation:
import numpy as np
from meridian.model import prior_distribution
mean = [1.0, 3.0] # Point estimate
std = [0.3, 2.0] # Standard error
# Define the LogNormal distribution
lognormal_dist = prior_distribution.lognormal_dist_from_mean_std(mean, std)
# Define the PriorDistribution object
prior = prior_distribution.PriorDistribution(roi_m=lognormal_dist)
To construct a LogNormal distribution from a range:
import numpy as np
from meridian.model import prior_distribution
low = [0.1, 0.5] # Range lower bound
high = [2.0, 10.0] # Range upper bound
mass_percent = 0.95 # Probability mass
# Define the LogNormal distribution
lognormal_dist = prior_distribution.lognormal_dist_from_range(
low, high, mass_percent=mass_percent
)
# Define the PriorDistribution object
prior = prior_distribution.PriorDistribution(roi_m=lognormal_dist)
However, if the results from previous experiments are near zero, you should
consider whether your prior beliefs are accurately represented by a
non-negative distribution such as the LogNormal distribution. We
highly recommend plotting the prior distribution to confirm it matches
your prior intuitions before proceeding with the analysis.