AB testing is an important part of agile software product development and revenue optimization nowadays. Optimizely claims to have a poweful solution for iOS, which is especially nice when you’re already using it for the good old web and want to support Android. Let’s have a closer look!

I won’t go into what’s AB testing in detail. Have look at wikipedia or google it. Basically it’s just splitting your customers into groups, distributing a different version of your website/app to each group and you measuring which variant performs better for your goals (revenue, interactions, …).

What does Optimizely for iOS offer?

With Optimizely you have a set of building blocks that you can use to compose your tests. You have ways to customize the variations, audience targeting possibilities and of course tracking of events and goals.

One nice thing of Optimizely, in comparsion to other AB testing tools like Amazon AB Testing, is that you can reuse your variation configuration options, goals, etc. for multiple tests. Of course it’s a problem if you run two tests at the same time with colliding configurations. E.g. test A wants the button to be red, but test B wants the button to be green. But Optimizely warns you about such conflicts.




To configure variations Optimizely you need to install the Optimizely SDK in you app and start the “Editor Mode”. In this “Editor Mode” your device is connected with the web interface of Optimizely and you can see the changes that you make directly on your device and a preview in the web interface. You have 3 ways to configure a variation:

1. UI Editor

With the UI editor you can modify your UI, e.g. button placement and the text of labels. Even as a non technical person. This is one of the larger features of Optimizely but also the one I, as a software engineer, don’t like and don’t trust at all. If you’ve ever done complex layouting and know about the problems of autolayout on iOS, it’s hard to trust an “automated engine” and your non-engineering product owner to cover all possible variants and edge cases of layouting - especially in times of differently sized iPhones. Luckily you can disable this feature with a simple boolean variable in your code ;-)

2. Live Variables

With Live Variables you can configure remote variables. You define the value in the Optimizely web interface and the app uses that value at runtime. In your iOS code you create a ‘placeholder’ with a default value. Optimizely registers this variable when you start the app in the “Editor Mode” and you can then use and configure it for your tests and variations. Back in your code you use the placeholder and the Optimizely SDK to get the current value for the user in respect to running AB tests on runtime. You can use all sorts of types for your variable: Strings, Numbers, Bools, CGRect, UIColor, …

3. Code Blocks

With Code Blocks you can define different blocks of code (yes simple objective blocks or swift closures) in your iOS code and configure which will be executed in the AB test variation. You cannot upload code via the Optimizely interface or change the code in any way after you submitted the app to Apple. Basically it’s just a glorified boolean(or number) variable to determine which case of an if-else should be executed.




You can customize which users should see a test with audience targeting using tags and a Universal User ID:

Tags

Tags are just key-value pairs which you can set on runtime for a specific user. It’s then possible to show a test only to users with certain tags. The tag matching queries are quite flexible. You can construct “and”, “or” and even partial match queries. For example you can define tags to target only your most active users or users who never bought anything etc..

We are even using tags for inhouse testing and simply defined a tag “BetaTesting” which is only set in our debug and inhouse builds. Targeting our AB test this way we can do both, avoid influencing live running AB tests with data from ourself and testing future AB tests in a more realistic environment than Optimizly’s built in preview mode.

Universal User ID

The Universal User ID allows you to deliver a consistent experience across devices for your users. You can hash any user identifying value and pass it to the SDK. Any user with the same Universal User ID will be put into the same bucket.




Of course an important part of AB testing is tracking which version performs better. Optimizely offers several tracking mechanisms:

Tap Goals/Events

You can track the click on certain elements without writing any code and just using the same mechanisms as the UI editor.

View Goals/Events

Optimizley can also fire an event when a view is shown to the user. You also set this up by just using the UI editor. I think this kind of tracking is a bit fuzzy - what counts as “view was shown”? iOS developers might know that it’s not always trivial to get notified when a certain view appears on screen so that the user can see it.

Custom Goals/Events

This is my preferred method. You can fire goals/events directly in your iOS code using the Optimizely SDK. Here you as a developer have the full control about when and how often the event will be fired.

Revenue Tracking

With revenue tracking it is possible to track the amount of money a user spent in the app - or any other relevant increasing number. Basically it’s just a counter with a name which you can increase by firing events. This is especially great if your test doesn’t have that one action that you want the user to do, but if you want to optimize retention or overall revenue.

You can use your existing tracking framework to fire events/goals, e.g. Google Analytics, but this requires you to use the native Google Analytics SDK, which might not be the case if you’re using an aggregator framework like Tealium.




Optimizely collects all the data for you and presents them in nice interactive graphs. You can see the usual statistics of triggered events for each variation, whether the test reached a statistical significance, conversion rate, difference interval, improvement in percent, …

They even have an option to export the data as csv - which Amazon AB Testing didn’t offer.

Evaluation

To wrap it up here’s a little summary with my personal pros and cons of Optimizely.

Pros:

  • It works cross platform
  • It is quite powerful
  • There’s a good and detailed documentation
  • You can edit running tests - this is especially helpful while developing and during the QA phase. Amazon AB testing requires you to create a copy every time you want to change something after starting the test once.
  • It’s possible to reset the recorded tracking data of the test (e.g. after you finished your local tests)
  • Direct support for partial rollouts: You can specify that only X % of your users participate in a test.

Cons:

  • You always need to connect a device/simulator with Optimizely (put your app into Edit Mode) to edit the parameters of the variation - even if you don’t use the UI editor and just want to edit the value of a live variable.
  • The UI editor is sometimes unreliable and even causes bugs in your app (sometimes even in preview mode). E.g. it hides the status bar and I even had a case where suddenly my table view cells had an incorrect size.
  • You cannot explicitly trigger the tracking event for “experiment started”. It happens implicitly when you first request the value of a live variable or similar.