As mentioned here, we’re seeing duplicate events in Kinesis following an upgrade to R103 (the issue is for R102 but after a test of R103, the result is the same).
We have 2 collectors (in ECS behind an LB) writing to “web_good” streams and 1 enricher (also in ECS) reading from “web_good” and writing to “enriched_good” (and “enriched_bad”). We’re seeing roughly half of all events being exact duplicates in “enriched_good”.
“web_good” has 2 shards. “enriched_good” has 4 shards.
I have tried completely restarting the ECS cluster as we have seen previously that some ECS tasks were not shut down completely during a redeploy due to issues with AWS’ ECS agent. It did not help.