We’re looking to optimize our costs and to re-shard the data-sink Kinesis queue (post-enricher) based on the realtime load. I know the enricher stores high-watermarks in DynamoDB - which is a complication for zero-downtime (or short downtime) resharding. Is this even possible?
We’re thinking of doing a variant of green/blue deploys whereby we’ll spin up a parallel enricher + kinesis queue + consumer lambda with the revised amount of shards, do a switch either in Elastic IP or ELB, let the old one drain out and take it out of commission. I was just wondering whether there’s a less heavy-handed approach. Let’s investigate this together!