I’m struggling to handle a peak of ~350k rpm of throughput during a soccer match transmission with Stream Collector/Kinesis. I don’t have much knowledge about akka and actors, but tried to increase parameters related to level of parallelism and changed for a c4.large ec2 instance type what gave more capacity. Another variable is the collector buffer size, and I ask if it affects only the instance -> kinesis flow but also the internet -> collector flow.
Today I observed two behaviors:
- Load Balancer -> Collector sending to Enrichment process on same instance -> Kinesis with enough shards
Load Ballancer -> Collector sending to Enrichment process on same instance -> stdout
2nd scenario handled 350k rpm with 6 instances behind Load Balancer.
1st scenario was a disaster (hundreds of thousands of 504 from Load Balancer), couldn’t even estimate exactly how many instances I should use to don’t get any 5xx responses. Important to highlight here that I did not have any error response from Kinesis (ex: max throughput rate/max records per secs).
How far is possible to go with Stream colletor and what is the maximum RPS or RPM that is possible to reach with a single instance by your experience?