The Meridian GeoX framework prioritizes a robust, data-driven approach to experimental design to minimize variance and advertiser budget requirements. Moving beyond standard randomized sampling, stratified sampling is proposed to generate design candidates which ensures that treatment and control groups are more comparable before an experiment begins. After stratified sampling, the design search process will filter design candidates and finalize designs that achieve the greatest statistical power.
Stratified sampling
Meridian GeoX uses a stratified sampling design to address the variance and high budget issues inherent in restricted sample sizes.
- Stratification process: It partitions geographical units (geos) into "strata," which are homogeneous clusters defined by shared characteristics including response volume, trend and seasonality.
- KPI featurization: Before clustering, KPI series are featurized using statistical metrics including the mean, coefficient of variation (CV), slope, autocorrelation (at lag 1), and dynamic time warping (DTW).
- Clustering and random assignment: A k-means algorithm can partition geos into strata based on these features. The default number of clusters is four. For each geo, randomized assignment into treatment or control groups is then performed within each cluster. Note that the random assignment also naturally expands to multi-cell study design.
Random sampling
Meridian GeoX supports random sampling, in which geo units are randomly partitioned to treatment and control groups without following the clustering structure at all. Random sampling is not generally recommended over the stratified sampling methodology, however, it becomes necessary in GeoX with a small number of geos.
Design search process
Meridian GeoX uses a rigorous design search pipeline to identify optimal experimental design candidates from a pool of likely designs generated from the stratified sampling method.
Evaluation metrics
The following metrics are evaluated under Meridian GeoX to closely monitor the quality of a design:
- Out-of-sample fit: Indicates whether or not the model fits a design, and is evaluated using the out-of-sample R-squared statistic to filter candidate designs.
- AA robustness: Validated through A/A simulations where p-values are checked to prevent statistically significant A/A biases.
Statistical power: Measured by the minimum detectable effect (MDE) of the relative lift. GeoX MDE lift percentage is independent of the options of analysis methodology, and is calculated using the formula:
$$MDE_{\,lift} = Min\:incremental\:conversion\:/\:Baseline\:control \:conversion\\= SE_{\,lift} \times (Z_{1-\alpha/2}\:+\:Z_{1-\beta})\:/\: Baseline\:control\:conversion$$In the formula, \( SE_{\,lift} \) denotes the standard error of absolute lift, while \( \alpha \) and \( 1-\beta \) represent the significance level and the target statistical power respectively. By default, a two-sided hypothesis testing is considered. \( Z \) corresponds to the standard normal quantiles.
Budget: Translates the statistical power into the fiscal requirement to reach power objectives.
Search pipeline
The design search process divides the entire pre-test data into initial training, validation, and final selection periods. The process then follows six sequential steps to finalize the designs. Refer to the diagram in this document for the sequential search pipeline steps:
