Our Snowplow open source pipeline runs on Google Kubernetes Engine, using PubSub for messages, and I’m wondering how each individual service can be scaled up horizontally, if that’s possible ?
Can each service be scaled horizontally without concurrency issue ?
Hi @Ltsrc ,
Kubernetes handles horizontal auto-scaling, and the strategy for scaling up up down depends on how you configure it.
Yes. PubSub consumers from a same subscription automatically share the consumption of the records, each record going to one consumer only.
Thanks for the reply !
That’s clear for the PubSub consumers services, but how about the collector ? I’m certain there is no issue scaling this service either, but I’d rather make sure now to avoid further issues