E2e Testing with Snowplow Mini/Micro

Hi there,

I’ve recently been looking at our options for running some integration/end-to-end tests of our snowplow trackers.

Snowplow Micro looks great, and the two recent blog posts with examples were very informative. However, our application’s current CI/CD process does not lend itself well to being spun up and integration tested during the test/build pipeline (It’s mostly just unit tests being run during the build).

We do however have e2e tests running against the deployed infrastructure in our lower (aws) environments. I would like to add some e2e tests (i.e. using a browser automation framework) around the snowplow tracking to this test suite.

FWIW - We already have snowplow mini up and running, which would normally be used for manual verification/QA during the development phase.

If we were to use snowplow micro in this setup, I would need to host it inside the VPC somewhere (so it is visible to the deployed application/services). My question is, is this defeating the purpose of micro? If we already have a mini instance deployed, can we instead use that?

Is there a (public) API endpoint for elasticsearch on the mini instance that I could request (e.g. from a running test) events from? I haven’t been able to figure this out yet (I’m not too familiar with the mini setup). In the usage guide it looks like the elastic search api might be private only. Is this correct?

In summary,
I’m wondering what my best option for e2e testing against (already deployed) trackers is: should I try to get mini to do what I want, or instead host a micro instance (on ECS/Fargate), either ephemerally or permanently.

Any suggestions or feedback is welcome!

Regards,
Dylan

Hi Dylan, welcome to the forums!

Yes - you’d need to ensure your application can access Micro either via a private IP (in the same IP) or over the internet using a public IP - either method should work fine. If you use a public IP micro doesn’t need to be in the same VPC as your application itself - though generally micro is expected to run within a CI environment alongside the applicaiton.

You can definitely use a mini instance if you want. The API isn’t as agreeable as rather than the micro API you have to use the Elasticsearch API but that is certainly possible (and is in fact what we use for the ‘Stream Live Data’ feature in the Snowplow Inspector.

It is public but it looks like the documentation isn’t rendering that code block correctly which I’ve flagged as an issue. It uses basic auth (so you’ll need to provide a username and password) and then you should be able to access the full Elasticsearch API via

curl --user username:password -XGET http://public-ip-of-mini-instance/elasticsearch

You can also access all the query / metadata APIs here if you want to do something like filtering on a specific app_id or user_id.

Hi Mike,

Thanks for the quick response.
This was returned after executing that curl request:

<a href="/elasticsearch/">Moved Permanently</a>.

I was also looking at this blog post by yali, but it’s from 2016 - so it might be out of date.

Which version are you running of Mini? This has changed a little from earlier versions (which needed to be accessed over port 9200) to a /elasticsearch endpoint.

Hey Mike,

Just to update you. The http://<public-ip-of-mini-instance>/elasticsearch endpoint does appear to be working for me now. I must have been doing something wrong, which I now can’t reproduce.

Thanks for your help :slight_smile:

1 Like