EmrEtlRunner Issues - taking too long on step 2

[alex]

@Kunal_Jain

byte-limit

Yes, see e.g. this thread for details:

Question on EmrEtlRunner options

Thanks for the additional detail @amitkhanal. I read that as you generating either 1,000 or 2,000 events per second, and you have the record limit in Kinesis S3 set to 500 records, so are going to be generating either 120 or 240 files a minute, 7,200 or 14,400 files per hour. This is going to massively overload not just EmrEtlRunner but also Hadoop. My recommendation is to: Change the buffer settings so you are generating only double-digit files per hour (the lower the better) Delete the fil…