Mojito on Snowplow: Open-source split testing framework

In 2012, I remember reading @yali & @alex 's blog posts introducing Snowplow Analytics - I was hooked on the slick server-less architecture and the excellent documentation, that I’ve been a user ever since. It’s with that excellent example Mint Metrics (who I work for) developed our own open-source project.

Earlier this week, we open-sourced our split testing framework Mojito which uses Snowplow heavily.

I think it’s of particular interest to Snowplowers because:

  • Snowplow Analytics is a first-class citizen for tracking/reporting on experiments (ahead of GA)
  • Feature parity with existing SaaS split testing tools (barring a WYSIWYG editor)
  • Lightweight JS library (<5kb vs >50kb for SaaS tools)
  • Native error-tracking and error handling for JS experiments on the web
  • Templateable Rmarkdown reports for tracking experiments with Snowplow data

It also offers a window into how we use Snowplow for measuring experiments (in case any other Snowplowers are using it).

We consciously decided to use Snowplow early on

SaaS split testing tools build their own event analytics tools and silo experiment data away form the rest of an organisation’s event stream. We find this results in data gaps and tracking black-boxes. Do VWO/Optimizely et al deduplicate events? What’s their bot exclusion like? etc. In a recent experiment we ran at Mint Metrics, GA and Snowplow counted just 30% of the test subjects tracked by the SaaS tool - most of the traffic was from bots.

Meanwhile as awesome as GA is, we disregarded it as an event analytics solution because of the aggressive sampling seen on our non-GA360 clients. Want sample data from the experiment sample groups? No thank you.

Snowplow’s open architecture and stability gave us confidence we’re measuring real events from users accurately. We can also drop it into clients’ sites at a moment’s notice and start building/launching experiments!

Error tracking & handling built-in & encouraged

All the experiments we build require custom JS - and lots of it. When you’re building entirely new features and pages in JS you’re bound to encounter errors…

So, why not track errors inside your variant code? You want your treatments to work, so we provide tools to track errors and report on them.

“Build software experiments better, together”

Clients sometimes mandate code review, formal QA and other practices before launching experiments. Unfortunately, building experiments inside WYSIWYG interfaces and expecting clients’ seasoned developers to review is a hard ask. Sometimes it’s slow, other times, it’s hard to communicate issues with the variant codes.

Instead, why not build / code-review and launch experiments from Github/Bitbucket?

With Mojito, this is how we designed it. Define your experiments in YAML & JS and let our CI tools build, test and publish the container:

id: test1
name: A/A test
trigger: trigger.js
sampleRate: 0.3
state: live
recipes:
  0:
    name: Control
  1:
    name: Treatment
    js: treatment.js

Then just add functions to trigger.js (for triggering the experiment at a desired event) and treatment.js (for transforming the page into a particular variant):

// trigger.js
function trigger(test) {
  if (document.location.pathname === '/') test.activate();
}

// treatment.js
function treatment() {
  console.log('You\'re now in the treatment group!');
}

Then using CI tools, we can publish experiments with Bitbucket Pipelines for CI or publish straight from the CLI using Gulp.

gulp scripts
gulp test # optionally define your own unit tests!
gulp publish

Snowplow data modelling & reports

Using Self-describing JSON events, we defined first-class events for exposures and errors. We also use Snowplow’s SQL runner for transforming our exposure events into tables for reporting.

Once the data is properly modelled (Redshift is the only supported storage target for now), we can produce reports on a series of metrics using Rmarkdown and Knitr.

wave_params <- list(
  client_id = "mintmetrics",
  wave_id = "ex1",
  start_date = "2019-05-15 09:19:45",
  stop_date = "2019-06-05 14:29:00",
  time_grain = "days",
  subject = "usercookie",
  recipes = c("Control", "Treatment")
)

goalList <- list(
  list(
    title = "Transactions",
    goal = "purchase",
    operand = "="
  ),
  list(
    title = "Page views",
    goal = "page_view %",
    operand = "like"
  )
)
goalList <- mojitoFullKnit(wave_params, goal_list = goalList)

Which produces a series of reports like this:

We’ve used it for over 500 experiments & millions of test exposures each month

Anyway, I’d be really keen to help other Snowplowers get set up with the Mojito Framework and track their events into their Snowplow data warehouses. For more details, you may want to see the original blog post or check out the github repos.

Drop us a comment on Github or below with any questions!

5 Likes

Mojito looks super-cool @robkingston! Love the Snowplow inspirations in the project. Really excited to see this grow…

1 Like

Thanks @alex!