EMR job won't start, doesn't allow --scale-down-behavior parameter


For some reason enrichment didn’t run last night. When I attempted re-running the EMR job this morning, Factotum would run the task but fail after a few minutes with the following output:

Task 'CF Enrich EMR' was started at 2017-05-23 23:35:23.368610459 UTC
Task 'CF Enrich EMR' stdout:
D, [2017-05-24T09:36:57.624000 #26469] DEBUG -- : Initializing EMR jobflow
Task 'CF Enrich EMR': failed after 2m, 5s. Reason: the task exited with a value not specified in continue_job - 4 (task expects one of the following return codes to continue [0])

Checking EMR’s latest jobs on AWS showed no record of the job starting.

Anyway, I decided to use the AWS CLI export found in the EMR web console, from a previous successful job:

When I ran the command, AWS threw an error:

Unknown options: --scale-down-behavior,TERMINATE_AT_TASK_COMPLETION

At this point, I just removed that parameter, and Enrichment has now started. Really odd that this has happened out of the blue!

Anyone else running into similar issues?


Just updating this thread with my solution.

  1. Pipeline wouldn’t run because I had shredded events that hadn’t yet been archived (rookie mistake)

  2. After updating system packages and upgrading AWS CLI to the latest version, it no longer threw this error:

Unknown options: --scale-down-behavior,TERMINATE_AT_TASK_COMPLETION