For some reason enrichment didn’t run last night. When I attempted re-running the EMR job this morning, Factotum would run the task but fail after a few minutes with the following output:
Task 'CF Enrich EMR' was started at 2017-05-23 23:35:23.368610459 UTC Task 'CF Enrich EMR' stdout: D, [2017-05-24T09:36:57.624000 #26469] DEBUG -- : Initializing EMR jobflow Task 'CF Enrich EMR': failed after 2m, 5s. Reason: the task exited with a value not specified in continue_job - 4 (task expects one of the following return codes to continue )
Checking EMR’s latest jobs on AWS showed no record of the job starting.
Anyway, I decided to use the AWS CLI export found in the EMR web console, from a previous successful job:
When I ran the command, AWS threw an error:
Unknown options: --scale-down-behavior,TERMINATE_AT_TASK_COMPLETION
At this point, I just removed that parameter, and Enrichment has now started. Really odd that this has happened out of the blue!
Anyone else running into similar issues?