AB Testing - Results

In Part 1 I talked about how we are adding interstitial ads into Picross HD. We had been getting tips and suggestions on how to do this in a way that doesn't drive our users crazy.  We decided to put together an AB Testing framework so we could learn from the results, and this is the results of that experiment.

Interstitial Ad Providers Test

One of the tests we conducted was on interstitial Ad providers.  We went with 2 different providers: ChartBoost and PlayHaven.  These ads are displayed on launch of the application and half our users got ChartBoost and the other half got PlayHaven ads.

Ads Shown

Even though ChartBoost showed only 2% more ads, it generated 8% more clicks.

Ad Clicks

In addition, it turns out ChartBoost also produced significantly more revenue.

I plan on continuing to work with PlayHaven to see if we can improve the results.  They have been great to work with, and I think they can do better.

The spike on the 29th was caused by two things at once.  The first was a one day promotion on AppsFire, and the second was we become more agressive on showing ads.

Agressive, Normal, or Lax Ads

Another experiment we wanted to run was to determine how often we should show ads.   We really didn't want to piss off our customers, and we weren't sure what the reaction would be. It's rather difficult introducing ads into a product that didn't have any ads to begin with.

We tested 3 different configurations: Agressive, Lax, and Normal.  Each configuration had about the same level of distribution.

"Time Between Shows" is the minimum time that has to pass before we show another ad.  For example, the agressive configuration won't show ads more then once every 5 minutes.  In the Lax case, we where only showing an ad every day.

We also didn't want the user's first experience in launching the app to be an ad.  So we used a "Sessions Before Active" configuration.  In the case of agressive, we require the app to have 2 sessions before showing an ad.

We then looked to see what effects this had on our app usage.  The Lax time was the closet measure we had to no ads, and it basically matched what we had seen before introducing ads.

Time in App with Lax Ad 

Our takeaway was that we weren't seeing a significant shift or loss of time spend in app.  Actually, you could argue the time spent in app actually improved a little bit.   

Time in App with Normal Ad 

We also found it interesting that there was no significant difference between Normal and Lax ads.

Time in App with Agressive Ad

Our AB Test framework allows us to adjust tests daily.  Using that capability, we pulled the Lax ads configuration on May 29th, and we may soon go to just showing Agressive Ads.  The one thing holding us back is sessions end almost 50% of the time after seeing an interstitial add!!

This is a bit concerning to us as we only do an InterstitialShow when a session starts. That means almost half our users are exiting the app right after they started the app.  Unfortunately, flurry doesn't let us track this by event so I can't tell how it compares across ad vendors.  I'm still not exactly sure what this really means for the app's performance as I haven't see it negatively effect the number of sessions or time spent in app.

Ad Feedback

We have had several negative responses to the ads.  However, it hasn't negatively effected our overall star rating or time spent in app.   Some of the complaints we received include:

• Too easy to miss click on the ad
• Video Ad consuming download bandwidth not appreciated (we also use AdColony for video ads).
• Ads being introduced in a paid for app.  Picross HD use to be a paid app a long time ago.

That last one was interesting because we don't show ANY still ads if any puzzle pack has been purchased, and we never show ads in the paid puzzle packs.  However, we show video ads in the "free" puzzle packs.   We are going to change this so that if any puzzle pack has been purchased we won't show any ads.  I think it was just too confusing for users the other way.

Conclusion

This has been an exciting experiment.  The data is telling us we should just go with agressive ads and use ChartBoost as our ad provider.  I'm still not comfortable pulling the agressive switch as I want to hear from more customers, and PlayHaven has been such a good group of people to work with I would like to see if they can perform better.

If we can make this model work, we would like to add more ad supported content in Picross HD.

Please feel free to follow me on twitter at @fivelakesstudio.  I would love to hear about your experiences with ads and A/B Testing.  Also feel free to contact me if you would be interesting in finding out more about our iOS AB Test framework.  

Thanks for reading and be sure to visit us at Five Lakes Studio.