|---| | [ View source on GitHub](https://github.com/google/meridian-geox/blob/main/meridian_geox/__init__.py) | ## Design Module [View source](https://github.com/google/meridian-geox/blob/main/meridian_geox/design.py) The design module specifies experiment parameters and generates optimized geographic splits (treatment versus control) based on historical data. The dataclasses that define input and output parameters and the primary functions to generate, compare, and visualize designs are listed in the sections that follow. ### DesignConfig A configuration class that defines the core parameters of a Meridian GeoX experiment. @dataclasses.dataclass class DesignConfig: experiment_duration: datetime.timedelta experiment_types: Union[ ExperimentType, dict[str, ExperimentType], ] = ExperimentType.HOLDBACK methodology: Methodology = Methodology.TBR geo_assignment_rule: GeoAssignmentRule = GeoAssignmentRule.STRATIFIED_SAMPLING cell_count: int = 1 alpha: float = 0.1 power: float = 0.8 test_type: TestType = TestType.TWO_SIDED design_output_count: int = 10 cost_per_incremental_conversion: Union[float, dict[str, float]] = 1.0 n_candidates: int = 100_000 n_ranked_candidates: int = 100 max_candidate_generation_retries: int = 10 seed: int = 42 slope_tolerance: float = 0.2 min_r2: float = 0.8 num_strata: int = 4 k_means_iterations: int = 10 | Attributes | Description | |---|---| | `experiment_duration` | Duration of the experiment, specified as a `datetime.timedelta`. Supported units `weeks` and `days`. | | `experiment_types` | Defines the nature of the test, such as holdback, go-dark, or heavy-up. For multi-cell experiments, different types can be assigned per cell using a dictionary. Otherwise, a single provided type is applied to all cells. | | `methodology` | The method chosen for design selection---for example, `TBR`. | | `geo_assignment_rule` | Rule used to assign geographical areas to different groups, such as `RANDOM` or `STRATIFIED_SAMPLING`. | | `cell_count` | Total number of treatment cells. Use `cell_count > 1` for multi-treatment scenarios sharing a common control. | | `alpha` | Significance level for the test. The default is 0.1 (90% confidence). | | `power` | Target statistical power (probability of detecting a true effect). The default is 0.8. | | `test_type` | The type of statistical test to be performed---for example, `ONE_SIDED` or `TWO_SIDED`. The default is `TWO_SIDED`. | | `design_output_count` | Number of ranked recommended designs to return. The default is 10. | | `cost_per_incremental_conversion` | Equivalent to *1* / *target iROAS* if revenue data is used. Used to estimate budget requirements for holdback experiments (cell). Optional (and ignored) for go-dark and heavy-up experiments (cell). For multi-cell experiments, different values can be assigned per holdback cell using a dictionary. If a single float is provided for a multi-cell design, it will be applied to all holdback cells. The default is 1.0. | **Advanced design search parameters** > [!CAUTION] > **Caution:** Modifying the default values of the advanced design search parameters is not recommended. | Attributes | Description | |---|---| | `n_candidates` | Number of candidates for the fast scoring step. The default is 100,000. | | `n_ranked_candidates` | Number of fully scored candidates. The default is 100. | | `max_candidate_generation_retries` | Maximum candidate generation retries. The default is 10. | | `seed` | Random number generator seed. The default is 42. | | `slope_tolerance` | Maximum allowed symmetric difference for slope check. The default is 0.2. | | `min_r2` | Minimum allowed R^2^ for the design. The default is 0.8. | | `num_strata` | Number of strata for stratified sampling. The default is 4. | | `k_means_iterations` | Number of iterations for k-means clustering. The default is 10. | ### Budget Budget constraint for a single cell. @dataclasses.dataclass class Budget: budget: Optional[float] = None budget_pct: Optional[float] = None | Attributes | Description | |---|---| | `budget` | The maximum budget for the experiment design (per cell), specified as a total budget amount. Should be specified for holdback cells. | | `budget_pct` | The maximum budget percentage change for the experiment design (per cell). Should be specified for go-dark and heavy-up cells. Must be negative for go-dark and positive for heavy-up. | ### Constraints Defines optional operational constraints for the design algorithm. @dataclasses.dataclass class Constraints: included_control_geos: Set[str] excluded_geos: Set[str] excluded_dates: Set[pd.Timestamp] budget_constraint: Union[Budget, dict[str, Budget], None] = None max_conversions_percent: Optional[float] = 0.3 | Attributes | Description | |---|---| | `included_control_geos` | Specific geographical areas that must be included in the control group. | | `excluded_geos` | Specific geographical areas to be excluded from the experiment design. | | `excluded_dates` | Specific dates to be left out of the experiment design. | | `budget_constraint` | The budget constraint for the experiment design (per cell). This could be a total budget amount or a budget percentage change. For multi-cell experiments, different values can be assigned per cell using a dictionary; otherwise, the single provided value is applied to all cells. If the budget percentage change is not specified for a go-dark cell, the default value is -100%. For heavy-up cell, the default value is 100%. | | `max_conversions_percent` | The maximum conversion volume allowed for the treatment group. For multi-cell designs, this percentage refers to the total for all treatment cells. The default is 0.3. | ### DesignSet A collection of generated experiment designs and their comparative metrics. @dataclasses.dataclass class DesignSet: designs: dict[str, Design] design_metrics: pd.DataFrame | Attributes | Description | |---|---| | `designs` | A dictionary mapping design IDs to individual [`Design`](https://developers.google.com/meridian/geox/api-reference#design) objects. | | `design_metrics` | A `DataFrame` containing ranked designs and their associated metrics like MDE and budget. | **Design metrics `DataFrame` columns** | Column | Description | |---|---| | `design_id` | Unique identifier for the design. | | `cell` | The treatment cell ID. | | `design_methodology` | The methodology used for the design. | | `r2` | Out of sample R^2^, calculated by : testing the model's predictive accuracy on historical data not used during the initial training phase. | | `mde` | Minimum Detectable Effect (MDE) represents the smallest uplift in the primary KPI that the experiment is powered to detect with statistical significance. A lower MDE indicates a more sensitive design. | | `mde_abs` | The minimum required incremental conversions. It is calculated as *`mde`* \* *`treatment_conversion_volume`*. | | `p_value (AA)` | The result of a robustness check where the model is applied to a period with no known treatment. A p-value greater than the significance level (typically 0.1) indicates that the design passes the A/A test and is not prone to false positives. | | `budget` | The projected total marketing spend changes needed. In go-dark or heavy-up studies, we calculate it based on the spend data and user input budget percent change. In holdback studies, we use CpIC to estimate the required budget. | | `design_implied_cpic` | The design-implied cost per incremental conversion (CpIC), calculated as the budget divided by the minimum required incremental conversions. | | `treatment_conversions_pct` | Treatment group conversion percentage. | | `treatment_geo_count` | Number of geos in treatment group. | ### Design Represents a single experiment design, including geo assignments and the configuration used. @dataclasses.dataclass class Design: designs: dict[str, PerCellDesign] control_geos: Set[str] excluded_geos: Set[str] excluded_dates: Set[pd.Timestamp] design_config: Optional[DesignConfig] = None constraints: Optional[Constraints] = None quality_check_result: Optional[QualityCheckResult] = None geo_stratum_labels: Optional[JnpArray] = None data: Optional[pd.DataFrame] = None def export_to_json(self) -> str @classmethod def load_from_json(cls, json_str: str) -> 'Design' | Attributes | Description | |---|---| | `designs` | A dictionary mapping treatment cell IDs to their respective `PerCellDesign` results. | | `control_geos` | The set of geographical areas assigned to the control group. | | `excluded_geos` | The set of geographical areas that were excluded from the experiment. Includes user manually excluded geos and outlier geos detected by data quality checks (if configured to be removed automatically). | | `excluded_dates` | Dates excluded from the design. Includes user manually excluded dates, and outlier dates detected by data quality checks (if configured to be removed automatically). | | `design_config` | The `DesignConfig` object used to create this specific design. | | `constraints` | The `Constraints` object applied during the design search. | | `quality_check_result` | Result of data quality checks performed on the input data. | | `geo_stratum_labels` | The stratum label of each geo, ordered by geo name. This is used for analysis. | | `data` | The data used for the design. This is used for analysis. | | **Method** | **Description** | |---|---| | `export_to_json` | Exports the design object to a JSON file. | | `load_from_json` | Loads a design object from a JSON file. | ### PerCellDesign Contains the specific assignments and statistical metrics for a single treatment cell within a design. @dataclasses.dataclass class PerCellDesign: treatment_geos: Set[str] minimum_detectable_effect: float design_implied_cpic: float p_value: float budget: float counterfactual_conversions: Optional[pd.DataFrame] = None | Attributes | Description | |---|---| | `treatment_geos` | The set of geographical areas assigned to the treatment group for this cell. | | `minimum_detectable_effect` | The smallest effect size the design is powered to detect. | | `design_implied_cpic` | The design-implied cost per incremental conversion (CpIC), calculated as the budget divided by the minimum required incremental conversions. | | `p_value` | The significance level of an A/A test verifying that the treatment and control groups are balanced before the experiment starts. | | `budget` | The estimated cost for this treatment cell based on the experiment parameters. | | `counterfactual_conversions` | Counterfactual conversion time series. Includes date, observed, and counterfactual conversions. This is used for plotting. | ### run_design() The primary function for generating and ranking potential experiment designs. def run_design( data: pd.DataFrame, design_config: DesignConfig, constraints: Constraints, data_quality_check_config: QualityCheckConfig = QualityCheckConfig() )-> DesignSet | Parameters | Description | |---|---| | `data` | Historical pretest time series containing `date`, `location`, `conversions`, and (optional) `spend`. | | `design_config` | Parameters for the experiment design. | | `constraints` | Operational constraints for the experiment design. | | `data_quality_check_config` | An option to configure automatic data quality checks. Default is to automatically remove geos with no response and outlier dates. | **Returns:** A [`DesignSet`](https://developers.google.com/meridian/geox/api-reference#designset) object containing a list of ranked [`Design`](https://developers.google.com/meridian/geox/api-reference#design) objects and associated metrics, such as MDE and budget. ### compare_designs() Compares multiple experiment designs across different methodologies, assignment rules, or configurations. def compare_designs( data: pd.DataFrame, design_requirements: list[tuple[DesignConfig, Constraints]], design_output_count: int = 10, ) -> DesignSet | Parameters | Description | |---|---| | `data` | Historical pretest time series. | | `design_requirements` | A list of tuples, each containing a [`DesignConfig`](https://developers.google.com/meridian/geox/api-reference#designconfig) and a [`Constraints`](https://developers.google.com/meridian/geox/api-reference#constraints) object. | | `design_output_count` | The number of designs to return. The default is 10. | **Returns:** A unified [`DesignSet`](https://developers.google.com/meridian/geox/api-reference#designset) containing the ranked designs from all provided configurations. ### concat_design_reports() Concatenates multiple [`DesignSet`](https://developers.google.com/meridian/geox/api-reference#designset) objects into a single ranked [`DesignSet`](https://developers.google.com/meridian/geox/api-reference#designset). def concat_design_reports( design_sets: list[DesignSet], design_output_count: int = 10 ) -> DesignSet | Parameters | Description | |---|---| | `design_sets` | A list of [`DesignSet`](https://developers.google.com/meridian/geox/api-reference#designset) objects to be merged. | | `design_output_count` | The number of top designs to return in the merged set. The default is 10. | **Returns:** A single [`DesignSet`](https://developers.google.com/meridian/geox/api-reference#designset) object containing all designs, re-ranked based on their metrics. ### plot_design() Generates visual representation of a specific design. def plot_design( design_to_plot: Design ) | Parameters | Description | |---|---| | `design_to_plot` | The specific [`Design`](https://developers.google.com/meridian/geox/api-reference#design) object to be visualized. | **Description:** Plots the time series of conversions, comparing the treatment group against the counterfactual to visualize the geo split effectiveness. ## Analysis Module [View source](https://github.com/google/meridian-geox/blob/main/meridian_geox/analysis.py) The analysis module calculates the incremental impact of a completed experiment using counterfactual modeling and robust inference. The dataclasses that define input and output parameters and the primary functions to generate and visualize experiment reports are listed in the following sections. ### AnalysisConfig Parameters required to execute a lift analysis for a completed GeoX study. @dataclasses.dataclass class AnalysisConfig: design: Design analysis_start_date: pd.Timestamp analysis_end_date: pd.Timestamp pretest_end_date: Optional[pd.Timestamp] = None excluded_dates: Set[pd.Timestamp] alpha: Optional[float] = None test_type: Optional[TestType] = None n_placebo_candidates: int = 100_000 n_top_placebos: int = 500 min_placebo_r2: float = 0.6 min_placebo_count_warning: int = 100 min_placebo_count_error: int = 10 | Attributes | Description | |---|---| | `design` | The specific geo-split and provenance information used during the design phase. | | `analysis_start_date` | Start date for analysis. | | `analysis_end_date` | End date for analysis, which may include a cooldown period. | | `pretest_end_date` | The end date of the pretest period. If not provided, the pretest period will be all dates before the `analysis_start_date`. | | `excluded_dates` | Specific dates to be excluded from the analysis , such as outlier days. | | `alpha` | Significance level. If omitted, it is inferred from the design config. | | `test_type` | The type of statistical test to be performed. If not provided, it will be inferred from the design config. | ### **Advanced analysis parameters** > [!CAUTION] > **Caution:** Modifying the default values of the advanced analysis parameters is not recommended. | Attributes | Description | |---|---| | `n_placebo_candidates` | Number of initial placebo candidates generated before selection. The default is 100,000. | | `n_top_placebos` | Number of top valid placebo candidates used for analysis. The default is 500. | | `min_placebo_r2` | Minimum out-of-sample R-squared score required for a placebo design to be kept for analysis. The default is 0.6. | | `min_placebo_count_warning` | Number of valid placebo candidates below which a warning will be logged. The default is 100. | | `min_placebo_count_error` | Number of valid placebo candidates below which an error will be raised. The default is 10. | ### AnalysisResult Contains the aggregated statistical output of a GeoX experiment across all cells. @dataclasses.dataclass class AnalysisResult: results: dict[str, AnalysisMetrics] analysis_config: AnalysisConfig excluded_geos: Set[str] excluded_dates: Set[pd.Timestamp] quality_check_result: Optional[QualityCheckResult] = None | Attributes | Description | |---|---| | `results` | A dictionary mapping each treatment cell to its corresponding [`AnalysisMetrics`](https://developers.google.com/meridian/geox/api-reference#analysismetrics) object. | | `analysis_config` | The configuration used for the analysis. | | `excluded_geos` | Geos excluded from the analysis. Includes all geos excluded during the design phase. | | `excluded_dates` | Dates excluded from the analysis. Includes user manually excluded dates from analysis config, and analysis phase outlier dates (if configured to be removed automatically). | | `quality_check_result` | Result of data quality checks performed on the input data. | ### AnalysisMetrics Contains metrics for a single cell analysis. @dataclasses.dataclass class AnalysisMetrics: lift: Estimate percent_lift: Estimate cumulative_lift: pd.DataFrame counterfactual_conversions: pd.DataFrame pointwise_difference: pd.DataFrame icpd: Optional[Estimate] = None cumulative_icpd: Optional[pd.DataFrame] = None descriptive_metrics: Optional[DescriptiveMetrics] = None | Attributes | Description | |---|---| | `lift` | The point estimate and confidence intervals for absolute incremental conversions. | | `percent_lift` | The estimated percentage lift with confidence intervals. | | `cumulative_lift` | The time series of incremental conversion lift estimates over the analysis period. | | `counterfactual_conversions` | Counterfactual conversion time series. Includes date, observed, counterfactual, and confidence intervals (test period only). | | `pointwise_difference` | Pointwise difference between observed and counterfactual conversions. Includes date, difference, and confidence intervals (test period only). | | `icpd` | Incremental conversion per dollar. Equivalent to iROAS if revenue data is used. Populated if spend data is available. | | `cumulative_icpd` | The time series of incremental conversion per dollar (iCPD) estimates over the analysis period. Only populated if spend data is available. | | `descriptive_metrics` | Descriptive metrics for a single cell analysis. This is used for Meridian integration. | ### Estimate An estimate with its confidence interval. @dataclasses.dataclass class Estimate: point_estimate: float lower_bound: float upper_bound: float standard_deviation: float p_value: float | Attributes | Description | |---|---| | `point_estimate` | The primary estimated value. | | `lower_bound` | The lower limit of the confidence interval. | | `upper_bound` | The upper limit of the confidence interval. | | `standard_deviation` | The standard deviation of the estimate. | | `p_value` | The significance level associated with the estimate. | ### DescriptiveMetrics Descriptive metrics for a single cell analysis. @dataclasses.dataclass class DescriptiveMetrics: estimated_bau_spend: Optional[float] = None | Attribute | Description | |---|---| | `estimated_bau_spend` | Represents the estimated BAU spend for the geos included in this cell's analysis (the specific cell's treatment geos plus control geos). It does not include spend from other treatment cells or non-experimental geos, and therefore does not represent the advertiser's total national spend. | ### analyze() Executes the lift analysis. def analyze( data: pd.DataFrame, analysis_config: AnalysisConfig, data_quality_check_config: QualityCheckConfig = QualityCheckConfig() )-> AnalysisResult | Parameters | Description | |---|---| | `data` | Full time series containing both pretest and test data for all geos. | | `analysis_config` | Configuration defining the methodology and time periods. | | `data_quality_check_config` | An option to configure automatic data quality checks. Default is to automatically remove outlier dates. | **Returns:** An [`AnalysisResult`](https://developers.google.com/meridian/geox/api-reference#analysisresult) object containing metrics for each treatment cell. ### plot_analysis() Generates visualization of the experiment analysis. def plot_analysis( analysis_result: AnalysisResult ) | Parameters | Description | |---|---| | `analysis_result` | The statistical output from the [`analyze()`](https://developers.google.com/meridian/geox/api-reference#analyze) function. | **Description:** Produces time series plots for counterfactual, pointwise difference, cumulative lift, and cumulative iCPD to visualize the estimated incremental impact for each treatment cell. ## Data Quality Module [View source](https://github.com/google/meridian-geox/blob/main/meridian_geox/data_quality/data_quality.py) ### QualityCheckConfig @dataclasses.dataclass class QualityCheckConfig: exclude_geos_no_response: bool = True exclude_outlier_dates: bool = True | Attributes | Description | |---|---| | `exclude_geos_no_response` | Determines whether to automatically exclude geos with no response during the design phase. Default is `True`. | | `exclude_outlier_dates` | Determines whether to automatically exclude outlier dates during the design or analysis phases. Default is `True`. | ### QualityCheckResult @dataclasses.dataclass class QualityCheckResult: quality_check_config: QualityCheckConfig quality_metrics: pd.DataFrame outlier_geos: Set[str] outlier_dates: Set[pd.Timestamp] | Attributes | Description | |---|---| | `quality_check_config` | The configuration settings used for the quality check. | | `quality_metrics` | A `DataFrame` containing detailed metrics resulting from the quality check. | | `outlier_geos` | The set of identified outlier geographical areas with no response. | | `outlier_dates` | The set of identified outlier dates detected during the quality check. | ### check_design_data_quality() def check_design_data_quality( data: pd.DataFrame, design_config: DesignConfig, quality_check_config: QualityCheckConfig ) -> QualityCheckResult **Description:** Checks the quality of the input data for the design phase. This check is integrated directly into the `run_design()` method, meaning data quality checks are automatically performed when generating designs. **Returns:** A [`QualityCheckResult`](https://developers.google.com/meridian/geox/api-reference#QualityCheckResult) object. ### check_analysis_data_quality() def check_analysis_data_quality( data: pd.DataFrame, analysis_config: AnalysisConfig, quality_check_config: QualityCheckConfig ) -> QualityCheckResult **Description:** Checks the quality of the input data for the analysis phase. This check is integrated directly into the `analyze()` method, meaning data quality checks are automatically performed during analysis. **Returns:** A [`QualityCheckResult`](https://developers.google.com/meridian/geox/api-reference#QualityCheckResult) object.