Consider the following generalizations about marketing mix modeling (MMM) as a
causal inference methodology:

- MMM is a causal inference tool for estimating the impact your advertising
  budget level and allocation have on KPI. MMM-derived insights such as ROI
  and response curves have a clear causal interpretation, and the modeling
  methodology must be appropriate for this type of analysis.

- The causal inference framework has important benefits, which are also
  critical components of any valid and interpretable MMM analysis:

  - ROI and other causal estimands are clearly defined using potential
    outcomes notation, which is both intuitive and mathematically rigorous.

  - Necessary assumptions can be determined and made transparent. All models
    require assumptions to provide valid estimates of the causal estimands.

- It is common knowledge that randomized experiments are considered the ideal
  way to estimate causal effects. MMM, however, is an example of causal
  inference from observational data.

- MMM has important advantages over experiments:

  - In the case of advertising, many experimental designs require individual
    user-level data that does not meet modern privacy standards. MMM uses
    observational data at an aggregate level that is privacy safe.

  - Experiments are often difficult to run due to cost and practicality.
    Observational data, on the other hand, is readily obtainable.

  - Experiments are typically designed to estimate one specific quantity. In
    advertising, for example, a geo experiment might be designed to estimate the
    ROAS of a specific channel such as TV. A causal inference model, such as
    MMM, can provide many insights such as ROI for every media channel, full
    response curves, and budget allocation without needing a complex and
    rigorous experimental design that might be impractical.

## Testable and untestable assumptions

Because MMM is based on observational data, it requires statistical assumptions
that are not necessary for most experiments. These assumptions can be
categorized as
[untestable](https://developers.google.com/meridian/docs/causal-inference/about-mmm-causal-inference-methodology#untestable_assumptions)
and
[testable](https://developers.google.com/meridian/docs/causal-inference/about-mmm-causal-inference-methodology#testable_assumptions).

Why do these assumptions matter from a practical standpoint? Multiple models can
have good fit and predictive power yet provide different ROI and optimization
results, therefore making it difficult to choose the best model.

### Untestable assumptions

- A condition known as conditional exchangeability is the main untestable
  assumption required for an MMM regression model to provide accurate causal
  inference results. This condition is untestable because there is no
  statistical way to determine its validity purely from observational data.

- Generally, *conditional exchangeability* means that the control variable set
  both:

  - Includes all confounding variables, which are variables that causally
    affect both media execution and KPI, and

  - Excludes any mediator variables, which are variables that lie in the
    causal pathway between media and KPI. (For a two-stage approach that
    allows including and modeling mediator variables representing brand
    equity, see [Full-Funnel MMM](https://developers.google.com/meridian/docs/advanced-modeling/full-funnel)).

- A causal graph can be used to justify the conditional exchangeability
  assumption. The causal graph must be constructed based on expert domain
  knowledge, as there is no statistical test to determine the correct graph
  structure purely from observational data.

- In reality, the exchangeability assumption is never perfectly met. The
  classic principle applies that "all models are wrong but some are useful".

### Testable assumptions

- Testable assumptions include anything related to the mathematical structure
  of the model. Consider:

  - How are media effects represented in the model, including lagged effects
    and diminishing returns?

  - How are control variables modeled? Are nonlinear transformations
    required?

  - How are trend and seasonality modeled?

- Testable assumptions can be evaluated to a certain extent by goodness of fit
  metrics, including prediction metrics such as out-of-sample R-squared.
  However:

  - Goodness of fit metrics don't give a complete picture of how good a
    model is for causal inference, and it is likely that the best model for
    causal inference is different from the best model for prediction.

  - The more candidate models you are comparing, the higher the risk of
    overfitting. For example, the best model is not the one that appears to
    have the best out-of-sample fit.

  - There is no threshold for R-squared or other metrics that makes a model
    good or bad. A model with 99% out-of-sample R-squared can still be a
    poor model for causal inference.

## Conclusions

There is no absolute best solution to MMM, which follows from the fundamental
principles of causal inference from observational data. We recommend that all
MMM practitioners think critically about MMM within a causal inference
framework, regardless of whether you use Meridian
or any other solution. The mission of Meridian is to provide you with
the utmost clarity about what your MMM is, how it works, and how you should
interpret your results.