Scala Stream Collector - scaling

Hi @Shin,

Pretty much this - though substitute “workers” for “servers.” You have one KCL instance (i.e. one Stream Enrich or similar) per server, but you may have more than one worker inside each KCL. You can have more workers than shards, but no more than one worker working on one shard at a time.

The whole thing is a bit more complicated than it should be - we have developed an in-house scaling and monitoring platform for real-time called Tupilak, which we hope to open-source later this year. We’ll do a preview post on this new tech (it’s pretty exciting) in a month or so…

2 Likes