Migrating from legacy drafts and experiments

Experiment and ExperimentArm have replaced CampaignExperiment. You can still use CampaignDraft to draft changes to a campaign and apply them all at once, but it will not be supported for experiment functionality.

Any experiment managed using CampaignExperiment can immediately be managed using Experiment instead. However, the reverse is not true. Once you begin managing an experiment using Experiment and ExperimentArm, the changes will not be written back to the CampaignExperiment and you may encounter errors trying to use CampaignExperiment from that point on.

Differences overview

It is recommended that you read through the complete documentation for the new experiments feature even if you're familiar with the previous version, because there are significant workflow differences. There is no new functionality available, however, the new interface will allow us to add further improvements to the feature in the future.

The overall flow of creating an experiment using Experiment and ExperimentArm is different from using the previous CampaignDraft and CampaignExperiment. Here is an overview of the differences:

  • You now create the Experiment first, and get an effective draft campaign ID from the ExperimentArm.
    • This differs from creating a CampaignDraft first and only getting a CampaignExperiment once the CampaignDraft was finalized.
  • You now create an ExperimentArm explicitly for your control arm, as well as for your treatment arm.
    • Previously, the "control arm" was implied from the base campaign for the CampaignDraft. This change allows us to more easily expand experiment functionality in the future.
  • Traffic split percent is now managed on the experiment arms, instead of the experiment.
    • Previously, you only specified the percentage for the experiment, and the remainder was reserved for the base campaign. Now, a percentage must be explicitly specified for each arm.
  • Draft campaigns are created automatically when creating a treatment arm. They are now called in_design_campaigns, and they mirror the campaigns you specified in the control arm. Once you have the draft campaign resource name, it functions identically to how it did in the past.