We decided to do a load testing exercise on our collector using avalanche. We used the following configuration for the simulation :
We ran the LinearPeak simulation. But as soon as we started the simulation the requests started to drop. We got the following error :
REQUEST Baseline 682 PageView Event 1479297981539 1479297981543 KO status.find.in(200,304,201,202,203,204,205,206,207,208,209), but actually found 503
After simulation ends we checked that total number of requests were around 9 million and around 5000 requests had above mentioned error.
But after running EMR and loading data into redshift we saw that only 0.13 million events came into table.
Our environment had autoscalling enabled and we were using m4.large type of EC2 instance.
What could be the reason that caused this problem?