July 15, 2022

A/B Tests

  1. There are a few use cases for A/B tests on Mediation.
    i. You can use A/B tests to test the hybrid monetization model (bidding plus waterfall) against pure waterfall monetization when you first start using Mediation.
    ii. Even after using Mediation for a while, if you want to test a new mediation network or test some different settings for line items, you can always use A/B tests.
  2. A/B tests are available at each Mediation placement level. You need to make sure that the basic bidding and waterfall are already set up for those Mediation placements.
  3. For the initial use case of testing hybrid vs waterfall, here is the best practice for setting up your first A/B test on Mediation
    i. Under the Mediation placement, you have finished setting up all bidding networks (including Chartboost, Meta Audience Network, AdColony, Tapjoy). You also have finished setting up the waterfall (including ad groups and line items).
    ii. Enable A/B test in Advanced Settings section of the placement.
    iii. Mediation dashboard will automatically create the A/B Test Settings section.
    iV. Control Group (A) is the existing bidding and waterfall setup for the Mediation placement.
    V. Test Group (B) is a copy of the existing setup. You should deactivate all the bidding networks and add the additional line items for the bidding networks (including Meta Audience Network, AdColony, Tapjoy) in the ad groups.
  4. Mediation
  5. automatically allocates 50% of ad requests to Control Group (A) and 50% of ad requests to Test Group (B).
    i. Each user is randomly assigned to one of the groups and is sticky to one group, which ensures the efficiency of A/B tests
  6. You can check A/B tests performance data in Analytics > A/B tests.
  7. If you find one of the groups is having better performance after reviewing A/B test analytics, you can disable the A/B test by disabling the worse performance group in the Mediation placement.
    i. All ad requests will automatically start going to the remaining active group once one group is being disabled.
    ii. You can not disable both groups. At least one group has to be active.
  8. If you want to re-activate the A/B test, you can directly edit any bidding and waterfall settings in the disabled group first and then turn on the “Active” toggle.