I’me trying to run an enrichment process using the emr-etl-runner from version r97.
The process is getting stuck trying to provision instances for master and core for just over one hour. Then it gets terminated, showing the below errors.
I’ve tried this multiple times altering:
The aws region (eu-west-1 and eu-west-2)
The type of instance m4.large and m4.2xlarge
Adjusted spot bidding to 50% over the prices shown by the AWS pricing page and to turn it off entirely.
The result is the same every time and I cannot see any log files or similar output beyond the above to pinpoint me to the issue.
Have you had the same configuration running in the past successfully?
It might also be worth reaching out to AWS support as they are often able to provide additional information on EMR internal failures. Often multiple causes (hitting EC2 quotas, certain network configurations etc) end up getting reported as ‘internal errors’.
Okay I identified the issue, I had the security group settings configured so that only the IP from which I triggreed the pracess was allowed in the routing table.
I"ve now opened this to 0.0.0.0/0 but is there a way to restrict it to what is actually needed?
