ABTesting

We are looking at adding interstitial ads into Picross HD.  We have been talking to people on the subject trying to get tips and suggestions on how to do this in a way that doesn't drive our users crazy and yet allows us to gain some additional revenue that we can use to keep making our games better.

One point that kept being made was to use AB Testing for figuring out how to optimize what we are trying to do.  This of course can be applied to many more things then ads, but I finally had a real need to solve this ABTesting problem.

The Problem

I wanted a simple ABTesting platform that would allow me to do the following:

1. Test different variables
2. Change variables without have to republish the App
3. Track / Report on the results

Variables

We use a simple XML file to define the test cases we want to experiment with.  Here is a simple example of the XML definition we use:

<ABTest>
   <testset name="AdTime">
      <casedef name="Normal" weight="33"/>
      <casedef name="Agressive" weight="33"/>
      <casedef name="Lax" weight="33"/>

      <vardef key="minTimeBetweenInterstitials">
         <case name="Normal" value="10.0"/>
         <case name="Agressive" value="5.0"/>
         <case name="Lax" value="1440"/>
      </vardef>

      <vardef key="numStartsBeforeShowingInterstitials">
         <case name="Normal" value="4"/>
         <case name="Agressive" value="2"/>
         <case name="Lax" value="10"/>
      </vardef>
   </testset>
</ABTest>

This was developed so that we could have multiple test sets and run multiple ABTests at the same time. The example above just defines a single test set.

With a test set, we can define multiple test cases we want to evaluate and we can define a frequency or weighting used to determine the percentage of users/devices that should be assigned that test case. In addition, each test set can have multiple variables and each variable can have values for each test case.

We wrote a simple objective-c class that manages the loading of the XML and provides a simple interface for reading variables from the ABTest.
   @interface FLABTestManager : NSObject
   + (FLABTestManager *)defaultManager;
   - (void)activateABTest;
   - (NSString *)stringForKey:(NSString *)key 
             withDefaultValue:(NSString *)defaultValue;
   - (int)intForKey:(NSString *)key withDefaultValue:(int)value;
   - (float)floatForKey:(NSString *)key withDefaultValue:(float)value;
   - (bool)boolForKey:(NSString *)key withDefaultValue:(bool)value;
   @end

The XML file is included as a project resource and then within the apps delegate method applicationDidBecomeActive you call activeABTest:
   - (void)applicationDidBecomeActive:(UIApplication *)application
   {
      ..
      [[FLABTestManager defaultManager] activateABTest];
      ..
   }

Once activated, the App will randomly pick a test case by weight for each test set defined.  It should be noted that once picked, the app will keep using the same test case even if the app quits.  This effectively locks a device into a particular test.  If the app is uninstalled and reinstalled a new test case may get chosen.

We choose to only activate the test cases from within applicationDidBecomeActive so we only activate once per session.  This is because it is possible to have a different configuration chosen during activation (see Changing Tests) and for reporting/tracking purposes we don't want to mix test cases (see Tracking/Reporting).

In order to read a variable under test, simple accessors methods are provided.  For example: If we where configured for Agressive Ad display and we had the XML definition above then the variable minTimeBetweenInterstitials would return "5.0":

[[FLABTestManager defaultManager] floatForKey:@"minTimeBetweenInterstitials"

                             withDefaultValue:@"10.0"];

We supply a default value for the case when the test hasn't been activated or if there was a problem determining the value.

Changing Tests

We wanted the ability to change tests without having to republish the App. This allows us to refine settings as we learn. In order to do this, a copy of the XML is stored and retrieved via a GoogleAppEngine service.

When an app doesn't have an AB Test loaded, it just uses the built in XML definition and a request is made to the web service for an updated version.  If there is an updated version, it will be downloaded and kept ready for the next ABTest activation.

When a new, downloaded, version of the ABTests are found the app will adjust to the new values accordingly.  For example if the apps current test case is no longer available in the XML definition, it will just pick a new test case.

Tracking / Reporting

Whenever activateABTest is called we log events to Flurry for the active test cases.  For example if the above XML definition is used and the device chooses Agressive for AdTime then the following two events would be logged:

   AdTime <== with parameter of Agressive
   AdTime.Agressive

This is enough information to allow us to create a segmentation on "AdTime.Agressive" so we can see how it compares with other tests.  We also log "AdTime" as an event by itself with the parameter set to the test case picked so we can validate the distribution of test cases.

Conclusion

We are still finishing testing on this new framework and we hope to submit to the AppStore next week. I think this has some nice potential to allow us to do more ABTesting on the work we do in order to improve the user experience of our products.  I will report back on our findings.

If you would be interesting in using this framework, just let me know.  I can't really hook you up with the GoogleAppEngine side of things, but I would be happy to share what I can on the iOS side.

Please feel free to follow me on twitter at @fivelakesstudio.  I would love to hear how others have solved this AB Testing issue.

I hope this was useful.  Please try out Picross HD and our other puzzle games for iOS.