Event driven architecture with a single Snowplow stream consumer

[af-tomwilkins]

I’m looking to set up an event driven architecture with the Snowplow real-time steam at the heart of it, having read this thread: How to use Snowplow as a Publish/Subscriber event bus?

The stream exists in a sub-account, and events are made available to the production account via Lambda and SQS. We chose this as we want to keep our solution serverless and scalable, but it does introduce the limitation of just one consumer.

The one consumer will be responsible for fanning events out to applications that care about them, so my question is, what would be the best way to maintain stability within our architecture? Our main concern is that if one application goes down, what happens to the event.

My current thinking is to have the single consumer publishing to SNS, and have SQS subscriptions attached with filter policies - but I’m very keen to hear if others have overcome this and/or have any recommendations.

Thanks!

[BenB]

Hi @af-tomwilkins ,

The one consumer will be responsible for fanning events

What kind of consumer is it ? Is it a standalone application ?

Our main concern is that if one application goes down, what happens to the event.

When consuming data from SQS, data should be acked only once its processing has been successful so that if consumer goes down in the middle of processing, data will be reprocessed when the consumer will be rebooted.

My current thinking is to have the single consumer publishing to SNS, and have SQS subscriptions

What about having one SQS queue between your single consumer and each of the applications, so that each app can be responsible for acking the events that it’s receiving ?

[af-tomwilkins]

Thanks for your reply

What kind of consumer is it ? Is it a standalone application ?

It’s a Lambda function and yes, it’s sole responsibility would be to read from the stream, parse events and distribute them.

What about having one SQS queue between your single consumer and each of the applications, so that each app can be responsible for acking the events that it’s receiving ?

This sounds similar to how I’m thinking it should work, but using SNS to handle the distribution which should make it easier to attach more consumers in future: