When creating a flow in Simon, you can run an A/B test on the selected segment. Experiments allow you to create multiple variants and select a percentage of the chosen segment to be held out as a control group. The different variants are then assigned to the triggered actions of the flow.

Configure an experiment

Clicking "Yes" under "Configure Testing (Optional)" opens the experiment configuration:

All contacts that are not assigned to the control group are a part of all triggered actions unless variant groups are defined. If you don't add any variants, there will only be a control group and a single "treatment" group.

Holdout group

Contacts that are selected for the holdout group will not be targeted by any actions you attach to the flow; this is the case whether or not you select multiple variants.

For example, if the action is adding contacts to a particular MailChimp Automation Workflow and holdout group size set to 10%, 10% of the contacts that are added to this flow will never be added to the selected Automation Workflow.

You can select any value between 0 and 100% for the holdout group size. As a best practice, a balanced control set of 50% can work well for a single-variant flow, and a holdout group of 10-20% can work well as a holdout in an experiment with multiple variants, or in one that's targeting a large number of contacts (more than 250,000).

Variants

Variants allow contacts in a segment to enter into differing actions. For example, contacts can receive sends that occur at different times of day or with different coupons amounts. Variants can be thought of as "treatment" groups.

Variants can be given a meaningful name in the free text field and removed by clicking the (x) to the right of the free text field.

Once you've named all the variants you want to use as part of an experiment, the next step is to assign the variants to actions in the flow.

Assigning actions to variants

If you've added variants to your experiment, you'll need to assign each action to a variant. (This step does not apply if you did not add any variants since each action is assumed to be part of the "treatment" group behind the scenes.) To do this, click on the "Select Variant" dropdown above an action and choose a variant. Note that if you try to save a flow that has variants before assigning your actions to them, you'll get an error. Now when you launch the flow, each action will only be applied to the contacts in the variant that it has been assigned.

Once you've added some actions and assigned each of them to a variant (for flows that have variants), you can save the flow.

Variant distribution

To understand how Simon partitions your contacts for an experiment, it's best to think "Holdout group first, then variants."

That is, we will first identify the control group, and prevent them from receiving any variant of the selected channel actions. If there are 1000 contacts in your flow with a holdout size of 10%, then 100 of them will receive no channel action, no matter how many variants are selected.

The contacts that are not selected for the control group are then distributed evenly between the variants. For example, if there are 3 variants in this flow with 1000 contacts and control size 10%, then 100 contacts will have no action assigned, 300 will be assigned to the first variant, 300 to the second, and 300 to the third.

If you haven't added any variants, Simon creates a "treatment" group behind the scenes that will execute all actions not in the control group when it runs.