Your inputs will be very helpful on this. As we are using snowplow heavily, and our use case demands sending events to both batch pipeline and real time pipeline separately. We will be sending ~100 million events everyday to each pipeline (i.e. 200 million event/day in total).
It would be great if you can help us with any workaround for this, also we would love to have this feature (allowing 2 emitters to send events to 2 different collectors) implemented from snowplow in their stack.
There one more question you can help me with. For real time events we are planning to use Kafka instead of Kinesis.
As the snowplow documentation of Snowplow 85 Metamorphosis says that Kafka support is in Beta :
- Is it still in Beta? Can we use this on production ?
- If yes, then how many companies (it would be great if you can name a few) using this on production?
These answers will make us move ahead with confidence on this implementation.