Robust inference

To determine whether the observed lift is statistically significant, Meridian GeoX is equipped with robust inference methods for current and future analysis methodology.

Meridian GeoX assignment mechanism

GeoX studies rely on highly constrained, mathematically optimized algorithms to determine geo assignment. Let \(Z \in \Omega = \lbrace 0, 1 \rbrace^N\) be the treatment assignment vector where \(N\) is the number of geo units included in the test, and let \(X\) be a matrix of pretest covariates. In stratified sampling, \(X\) includes historical conversion volumes, trends and seasonality patterns. The design algorithm typically applies a constraint function or minimizes a cost function \(C(Z, X)\) to yield a heavily restricted assignment:

$$Z^* = \arg\min_{Z \in \Omega} C(Z, X)$$

You don't need to select the exact minimum, as it's quite likely trapped in an over-fitting solution. Alternatively, it's enough to uniformly select geo assignment \(Z\) from a valid subset of acceptable designs:

$$\Omega_D = \{Z \in \Omega \mid C(Z, X) \le \epsilon\}$$

The inference problem

To evaluate the treatment effect using causal estimators (for example, time-based regression (TBR)), the analysis tests the sharp null hypothesis \(H_0:Y_i(1)=Y_i(0)\) for each treated unit \(i\). In exact randomization inference, the p-value represents the probability of observing a test statistic \(T\) at least as extreme as the observed value, computed across the distribution of all possible assignments under the actual assignment mechanism.

Advantages of design-aware inference

The design-aware inference is designed to objectively evaluate the true significance of incrementality findings in GeoX. This approach solves the common inference challenge that many conventional inference methodologies (such as the t-test or sliding-window test) frequently fail to control the false discovery rate. Compared to these methods, design-aware inference is much less likely to detect false significant incrementality effects. Detailed methodology and findings can be found in our research paper.

Design-aware inference is also superior to standard placebo methods because it resolves a fundamental mismatch in how geo experiments are evaluated. Standard methods assume test geos were selected completely at random, while most GeoX studies rely on highly constrained, mathematically optimized algorithms to determine geo assignment.

How design-aware inference addresses the specific limitations of standard methods:

• The standard placebo testing draws assignment \(Z\) uniformly from \(Ω\) (by shuffling treatment labels) to build an empirical null distribution that reflects noise in the estimator. Note that in placebo tests, both \(Z\) and \(Ω\) are restricted to using control geos only. However, because the standard test ignores the study's design constraints, the procedure frequently generates "bad splits", such as comparing a treatment group of exclusively high-volume large metros against a control group of low-volume smaller geos. These unmatchable splits pollute the empirical null distribution with massive errors. This inaccuracy doesn't just inflate the type-I error rate, it also makes detecting a true incremental lift more difficult.

• Design-aware inference restricts placebo generation to the exact topology of the original experimental design by drawing assignment \(Z\) uniformly from \(\Omega_D\). Placebos are generated using the identical objective functions, budget limits, and design constraints. Therefore, every placebo is a valid, high-quality split, the null distribution perfectly reflects the true baseline uncertainty of the inference problem. Compared to the standard approach, this structural tightening of the null distribution significantly reduces the minimum detectable effect (MDE), which increases your statistical power.

Implement design-aware inference

Implementing design-aware inference necessitates a formal synchronization between the design phase and the subsequent analysis.

The following sections lay out the steps to implement design-aware inference.

Formalize the assignment mechanism: \(P(Z \mid X)\)

The optimization problem translates your design configurations into a programmable constraint function. Assuming an optimization heuristic, such as greedy matching, is utilized to assign units, let \(\pi(\cdot)\) represent a design algorithm that generates the assignment \(Z\) by uniformly sampling from \(\Omega_D\).

Design-aware inference requires access to the generative algorithm \(\pi(\cdot)\) or the cached subspace \(\Omega_D\).

Generate design-aware placebo vectors

Rather than relying on uniform shuffling, the placebo matrix is constructed by drawing directly from \(\Omega_D\).

1. Initialize the original pre-period covariate matrix \(X\) for all control units.
2. Run the design algorithm \( \pi(X) \) identically to the experiment setup phase, using unique random seeds to explore the space of valid designs.
3. Collect \(K\) placebo assignments \(\{Z^{(1)}, Z^{(2)},\dots,Z^{(K)}\}\) such that every \(Z^{(k)}\) adheres to structural constraints, including stratification, budget limits, and pretest quality filtering.

Compute the empirical null distribution

To estimate the causal effect for each valid placebo assignment \(Z^{(k)}\), follow these steps:

1. Fit the counterfactual model using only pretest data on assignment \(Z^{(k)}\).
2. Predict the counterfactual outcome for the treated units defined in \(Z^{(k)}\) during the actual test phase.
3. Calculate the placebo test statistic \(T^{(k)}\), which is the total incremental effect on the treated units defined in \(Z^{(k)} : T^{(k)} = \sum_{t > t_0} (Y_t^{(k)} - \widehat{Y_t^{(k)}(0)})\).

Because each \(Z^{(k)}\) is a high-quality split by design, the estimator yields robust synthetic matches of the observed effect \(T_{\text{obs} }\) in real design. The resulting distribution of \(T^{(k)}\) represents the level of uncertainty under the sharp null hypothesis.

Calculate p-value and confidence intervals (valid exact inference)

Under the sharp null hypothesis, the p-value is the probability that a placebo split produces a test statistic as extreme as the observed effect \(T_{\text{obs}}\). To account for variations in the baseline model fit, you can use the studentized test statistic instead. The studentized test statistic is defined as the ratio of the post-period effect to the pre-period root mean squared error (RMSE):

$$\tilde{T}^{(k)} = \frac{T^{(k)}}{RMSE_{\text{pre}}^{(k)}}$$

The refined p-value calculation is defined as:

$$p = \frac{1}{K+1} \left\{ 1 + \sum_{k=1}^K \mathbb{I} \left( \tilde{T}^{(k)} \text{ is more extreme than } \tilde{T}_{\text{obs}} \right) \right\}$$

The confidence interval of the incremental effect can be subsequently retrieved from the inversion of these p-values.