Snowplow Mini - Two Kinesis sinks to Elasticsearch?

For engineers Snowplow Mini
1

Hi,

Thanks for your help. Can we have two snowplow-minis connecting to one elasticsearch storage? i.e., two kinesis-elasticsearch-sinks from two different snowplow-minis moving data to one remote elasticsearch cluster? Kindly help. We wanted the failover set up for collectors, hence thinking of this route.

Thanks,
Manju

2

Hi @manju,

Yes, you can. The question is why - your setup looks a bit weird. Why not to use load balanced collectors without additional load. Than setup rest of stack on additional machine. Interconnections with Kinesis/Kafka. IMHO collectors should be as HR/HA as possible. Any extra load would make me sleepless…

2 Likes
3

Hey @manju,

To expand on this I would just like to highlight the fact that Snowplow Mini is not a production ready setup. It is designed and meant only for low-scale testing and nothing more.

For production I would recommend having a read of this post which highlights the lambda architecture: How to setup a Lambda architecture for Snowplow

4

Thank you both!
@grzegorzewald,
Sorry, can you please explain how best can this be done? I am using elastic load balancer to convert https to http. I have nginx router watching two collectors (in two different snowplow-minis) for failovers. Should I pass data from the collector to a single enrichment stream?

If there is a better way this could be done, please let me know. If possible we do not want to use Kinesis Shards.

@josh, thank you for the info on Lambda architecture.