To determine whether the observed lift is statistically significant, Meridian
GeoX is equipped with robust inference methods for current and future analysis
methodology.

## Meridian GeoX assignment mechanism

GeoX studies rely on highly constrained, mathematically optimized algorithms to
determine geo assignment. Let \\(Z \\in \\Omega = \\lbrace 0, 1 \\rbrace\^N\\) be
the treatment assignment vector where \\(N\\) is the number of geo units
included in the test, and let \\(X\\) be a matrix of pretest covariates. In
stratified sampling, \\(X\\) includes historical conversion volumes, trends
and seasonality patterns. The design algorithm typically applies a constraint
function or minimizes a cost function \\(C(Z, X)\\) to yield a heavily
restricted assignment:
$$Z\^\* = \\arg\\min_{Z \\in \\Omega} C(Z, X)$$

You don't need to select the exact minimum, as it's quite likely trapped in an
over-fitting solution. Alternatively, it's enough to uniformly select geo
assignment \\(Z\\) from a valid subset of acceptable designs:
$$\\Omega_D = \\{Z \\in \\Omega \\mid C(Z, X) \\le \\epsilon\\}$$

## The inference problem

To evaluate the treatment effect using causal estimators (for example,
time-based regression (TBR)), the analysis tests the sharp null hypothesis
\\(H_0:Y_i(1)=Y_i(0)\\) for each treated unit \\(i\\). In exact
randomization inference, the p-value represents the probability of observing a
test statistic \\(T\\) at least as extreme as the observed value, computed
across the distribution of all possible assignments under the actual assignment
mechanism.

## Advantages of design-aware inference

The design-aware inference is designed to objectively evaluate the true
significance of incrementality findings in GeoX. This approach solves
the common inference challenge that many conventional inference methodologies
(such as the t-test or sliding-window test) frequently fail to control the false
discovery rate. Compared to these methods, design-aware inference is much less
likely to detect false significant incrementality effects. Detailed methodology
and findings can be found in our
[research paper](https://research.google/pubs/pub1090193/).

Design-aware inference is also superior to standard placebo methods because it
resolves a fundamental mismatch in how geo experiments are evaluated. Standard
methods assume test geos were selected completely at random, while most GeoX
studies rely on highly constrained, mathematically optimized algorithms to
determine geo assignment.

How design-aware inference addresses the specific limitations of standard
methods:

- The standard placebo testing draws assignment \\(Z\\) uniformly from \\(Ω\\)
  (by shuffling treatment labels) to build an empirical null distribution that
  reflects noise in the estimator. Note that in placebo tests, both \\(Z\\) and
  \\(Ω\\) are restricted to using control geos only. However, because the standard
  test ignores the study's design constraints, the procedure frequently generates
  "bad splits", such as comparing a treatment group of exclusively high-volume
  large metros against a control group of low-volume smaller geos. These
  unmatchable splits pollute the empirical null distribution with massive errors.
  This inaccuracy doesn't just inflate the type-I error rate, it also makes
  detecting a true incremental lift more difficult.

- Design-aware inference restricts placebo generation to the exact topology of
  the original experimental design by drawing assignment \\(Z\\) uniformly from
  \\(\\Omega_D\\). Placebos are generated using the identical objective
  functions, budget limits, and design constraints. Therefore, every placebo is a
  valid, high-quality split, the null distribution perfectly reflects the true
  baseline uncertainty of the inference problem. Compared to the standard
  approach, this structural tightening of the null distribution significantly
  reduces the minimum detectable effect (MDE), which increases your statistical
  power.

## Implement design-aware inference

Implementing design-aware inference necessitates a formal synchronization
between the design phase and the subsequent analysis.

The following sections lay out the steps to implement design-aware inference.

## Formalize the assignment mechanism: \\(P(Z \\mid X)\\)

The optimization problem translates your design configurations into a
programmable constraint function. Assuming an optimization heuristic, such as
greedy matching, is utilized to assign units, let \\(\\pi(\\cdot)\\) represent a
design algorithm that generates the assignment \\(Z\\) by uniformly sampling
from \\(\\Omega_D\\).

Design-aware inference requires access to the generative algorithm
\\(\\pi(\\cdot)\\) or the cached subspace \\(\\Omega_D\\).

## Generate design-aware placebo vectors

Rather than relying on uniform shuffling, the placebo matrix is constructed by
drawing directly from \\(\\Omega_D\\).

1. Initialize the original pre-period covariate matrix \\(X\\) for all control units.
2. Run the design algorithm \\( \\pi(X) \\) identically to the experiment setup phase, using unique random seeds to explore the space of valid designs.
3. Collect \\(K\\) placebo assignments \\(\\{Z\^{(1)}, Z\^{(2)},\\dots,Z\^{(K)}\\}\\) such that every \\(Z\^{(k)}\\) adheres to structural constraints, including stratification, budget limits, and pretest quality filtering.

## Compute the empirical null distribution

To estimate the causal effect for each valid placebo assignment \\(Z\^{(k)}\\),
follow these steps:

1. Fit the counterfactual model using only pretest data on assignment \\(Z\^{(k)}\\).
2. Predict the counterfactual outcome for the treated units defined in \\(Z\^{(k)}\\) during the actual test phase.
3. Calculate the placebo test statistic \\(T\^{(k)}\\), which is the total incremental effect on the treated units defined in \\(Z\^{(k)} : T\^{(k)} = \\sum_{t \> t_0} (Y_t\^{(k)} - \\widehat{Y_t\^{(k)}(0)})\\).

Because each \\(Z\^{(k)}\\) is a high-quality split by design, the estimator
yields robust synthetic matches of the observed effect \\(T_{\\text{obs} }\\)
in real design. The resulting distribution of \\(T\^{(k)}\\) represents the
level of uncertainty under the sharp null hypothesis.

## Calculate p-value and confidence intervals (valid exact inference)

Under the sharp null hypothesis, the p-value is the probability that a placebo
split produces a test statistic as extreme as the observed effect
\\(T_{\\text{obs}}\\). To account for variations in the baseline model fit, you
can use the studentized test statistic instead. The studentized test statistic
is defined as the ratio of the post-period effect to the pre-period root mean
squared error (RMSE):
$$\\tilde{T}\^{(k)} = \\frac{T\^{(k)}}{RMSE_{\\text{pre}}\^{(k)}}$$

The refined p-value calculation is defined as:
$$p = \\frac{1}{K+1} \\left\\{ 1 + \\sum_{k=1}\^K \\mathbb{I} \\left( \\tilde{T}\^{(k)} \\text{ is more extreme than } \\tilde{T}_{\\text{obs}} \\right) \\right\\}$$

The confidence interval of the incremental effect can be subsequently retrieved
from the inversion of these p-values.