Snowplow ETL on AWS EMR hangs


#1

We have an ETL that hangs for about 3 weeks on Enrich stage. Usually takes about a minute for the whole cluster to run.

controller log of the cluster step:

    2018-10-04T04:36:29.999Z INFO Ensure step 3 jar file command-runner.jar
    2018-10-04T04:36:29.999Z INFO StepRunner: Created Runner for step 3
    INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar spark-submit --class com.snowplowanalytics.snowplow.enrich.spark.EnrichJob --master yarn --deploy-mode cluster s3://snowplow-hosted-assets-us-west-2/3-enrich/spark-enrich/snowplow-spark-enrich-1.14.0.jar --input-format clj-tomcat --etl-timestamp 1538627422352 --iglu-config ewogICJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3MuaWdsdS9yZXNvbHZlci1jb25maWcvanNvbnNjaGVtYS8xLTAtMSIsCiAgImRhdGEiOiB7CiAgICAiY2FjaGVTaXplIjogNTAwLAogICAgInJlcG9zaXRvcmllcyI6IFsKICAgICAgewogICAgICAgICJuYW1lIjogIklnbHUgQ2VudHJhbCIsCiAgICAgICAgInByaW9yaXR5IjogMCwKICAgICAgICAidmVuZG9yUHJlZml4ZXMiOiBbICJjb20uc25vd3Bsb3dhbmFseXRpY3MiIF0sCiAgICAgICAgImNvbm5lY3Rpb24iOiB7CiAgICAgICAgICAiaHR0cCI6IHsKICAgICAgICAgICAgInVyaSI6ICJodHRwOi8vaWdsdWNlbnRyYWwuY29tIgogICAgICAgICAgfQogICAgICAgIH0KICAgICAgfQogICAgXQogIH0KfQo= --enrichments eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9lbnJpY2htZW50cy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6W3sic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvcmVmZXJlcl9wYXJzZXIvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsibmFtZSI6InJlZmVyZXJfcGFyc2VyIiwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJpbnRlcm5hbERvbWFpbnMiOltdfX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvdWFfcGFyc2VyX2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3ciLCJuYW1lIjoidWFfcGFyc2VyX2NvbmZpZyIsImVuYWJsZWQiOnRydWUsInBhcmFtZXRlcnMiOnt9fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9hbm9uX2lwL2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7Im5hbWUiOiJhbm9uX2lwIiwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJhbm9uT2N0ZXRzIjoxfX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvaXBfbG9va3Vwcy9qc29uc2NoZW1hLzItMC0wIiwiZGF0YSI6eyJuYW1lIjoiaXBfbG9va3VwcyIsInZlbmRvciI6ImNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdyIsImVuYWJsZWQiOnRydWUsInBhcmFtZXRlcnMiOnsiZ2VvIjp7ImRhdGFiYXNlIjoiR2VvTGl0ZTItQ2l0eS5tbWRiIiwidXJpIjoiaHR0cDovL3Nub3dwbG93LWhvc3RlZC1hc3NldHMuczMuYW1hem9uYXdzLmNvbS90aGlyZC1wYXJ0eS9tYXhtaW5kIn19fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jdXJyZW5jeV9jb252ZXJzaW9uX2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJlbmFibGVkIjpmYWxzZSwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwibmFtZSI6ImN1cnJlbmN5X2NvbnZlcnNpb25fY29uZmlnIiwicGFyYW1ldGVycyI6eyJhY2NvdW50VHlwZSI6IkRFVkVMT1BFUiIsImFwaUtleSI6Int7S0VZfX0iLCJiYXNlQ3VycmVuY3kiOiJVU0QiLCJyYXRlQXQiOiJFT0RfUFJJT1IifX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cuZW5yaWNobWVudHMvc3FsX3F1ZXJ5X2VucmljaG1lbnRfY29uZmlnL2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7InZlbmRvciI6ImNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cyIsIm5hbWUiOiJzcWxfcXVlcnlfZW5yaWNobWVudF9jb25maWciLCJlbmFibGVkIjpmYWxzZSwicGFyYW1ldGVycyI6eyJpbnB1dHMiOlt7InBsYWNlaG9sZGVyIjoxLCJwb2pvIjp7ImZpZWxkIjoidXNlcl9pZCJ9fSx7InBsYWNlaG9sZGVyIjoxLCJqc29uIjp7ImZpZWxkIjoiY29udGV4dHMiLCJzY2hlbWFDcml0ZXJpb24iOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jbGllbnRfc2Vzc2lvbi9qc29uc2NoZW1hLzEtKi0qIiwianNvblBhdGgiOiIkLnVzZXJJZCJ9fSx7InBsYWNlaG9sZGVyIjoyLCJwb2pvIjp7ImZpZWxkIjoiYXBwX2lkIn19XSwiZGF0YWJhc2UiOnsicG9zdGdyZXNxbCI6eyJob3N0IjoiY2x1c3RlcjAxLnJlZHNoaWZ0LmFjbWUuY29tIiwicG9ydCI6NTQzOSwic3NsTW9kZSI6dHJ1ZSwidXNlcm5hbWUiOiJzbm93cGxvd19lbnJpY2hfcm8iLCJwYXNzd29yZCI6IjFhc0lrSmVkIiwiZGF0YWJhc2UiOiJjcm0ifX0sInF1ZXJ5Ijp7InNxbCI6IlNFTEVDVCB1c2VybmFtZSwgZW1haWxfYWRkcmVzcywgZGF0ZV9vZl9iaXJ0aCBGUk9NIHRibF91c2VycyBXSEVSRSB1c2VyID0gPyBBTkQgY2xpZW50ID0gPyBMSU1JVCAxIn0sIm91dHB1dCI6eyJleHBlY3RlZFJvd3MiOiJBVF9NT1NUX09ORSIsImpzb24iOnsic2NoZW1hIjoiaWdsdTpjb20uYWNtZS91c2VyL2pzb25zY2hlbWEvMS0wLTAiLCJkZXNjcmliZXMiOiJBTExfUk9XUyIsInByb3BlcnR5TmFtZXMiOiJDQU1FTF9DQVNFIn19LCJjYWNoZSI6eyJzaXplIjozMDAwLCJ0dGwiOjYwfX19fSx7InNjaGVtYSI6ImlnbHU6Y29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93L2V2ZW50X2ZpbmdlcnByaW50X2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3ciLCJuYW1lIjoiZXZlbnRfZmluZ2VycHJpbnRfY29uZmlnIiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJleGNsdWRlUGFyYW1ldGVycyI6WyJlaWQiLCJudWlkIiwic3RtIiwiY3YiXSwiaGFzaEFsZ29yaXRobSI6Ik1ENSJ9fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cy9odHRwX2hlYWRlcl9leHRyYWN0b3JfY29uZmlnL2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7InZlbmRvciI6ImNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cyIsIm5hbWUiOiJodHRwX2hlYWRlcl9leHRyYWN0b3JfY29uZmlnIiwiZW5hYmxlZCI6ZmFsc2UsInBhcmFtZXRlcnMiOnsiaGVhZGVyc1BhdHRlcm4iOiIuKiJ9fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cy93ZWF0aGVyX2VucmljaG1lbnRfY29uZmlnL2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7ImVuYWJsZWQiOmZhbHNlLCJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cuZW5yaWNobWVudHMiLCJuYW1lIjoid2VhdGhlcl9lbnJpY2htZW50X2NvbmZpZyIsInBhcmFtZXRlcnMiOnsiYXBpS2V5Ijoie3tLRVl9fSIsImNhY2hlU2l6ZSI6NTEwMCwiZ2VvUHJlY2lzaW9uIjoxLCJhcGlIb3N0IjoiaGlzdG9yeS5vcGVud2VhdGhlcm1hcC5vcmciLCJ0aW1lb3V0Ijo1fX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvY29va2llX2V4dHJhY3Rvcl9jb25maWcvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsibmFtZSI6ImNvb2tpZV9leHRyYWN0b3JfY29uZmlnIiwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJjb29raWVzIjpbInNwIl19fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9qYXZhc2NyaXB0X3NjcmlwdF9jb25maWcvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwibmFtZSI6ImphdmFzY3JpcHRfc2NyaXB0X2NvbmZpZyIsImVuYWJsZWQiOmZhbHNlLCJwYXJhbWV0ZXJzIjp7InNjcmlwdCI6IlpuVnVZM1JwYjI0Z2NISnZZMlZ6Y3lobGRtVnVkQ2tnZXcwS0RRb2dJSFpoY2lCd2JHRjBabTl5YlNBOUlHVjJaVzUwTG1kbGRGQnNZWFJtYjNKdEtDa3NEUW9nSUNBZ0lDQmhjSEJKWkNBZ0lDQTlJR1YyWlc1MExtZGxkRUZ3Y0Y5cFpDZ3BPdzBLRFFvZ0lHbG1JQ2h3YkdGMFptOXliU0E5UFNBaWMyVnlkbVZ5SWlBbUppQmhjSEJKWkNBaFBTQWljMlZqY21WMElpa2dldzBLSUNBZ0lIUm9jbTkzSUNKVFpYSjJaWEl0YzJsa1pTQmxkbVZ1ZENCb1lYTWdhVzUyWVd4cFpDQmhjSEJmYVdRNklDSWdLeUJoY0hCSlpEc05DaUFnZlEwS0lDQU5DaUFnYVdZZ0tHRndjRWxrSUQwOUlHNTFiR3dwSUhzTkNpQWdJQ0J5WlhSMWNtNGdXMTA3RFFvZ0lIME5DZzBLSUNBdkx5QlZjMlVnYm1WM0lGTjBjbWx1WnlncElHSmxZMkYxYzJVZ2FIUjBjRG92TDI1bGJITnZibmRsYkd4ekxtNWxkQzh5TURFeUx6QXlMMnB6YjI0dGMzUnlhVzVuYVdaNUxYZHBkR2d0YldGd2NHVmtMWFpoY21saFlteGxjeThOQ2lBZ2RtRnlJR0Z3Y0Vsa1ZYQndaWElnUFNCdVpYY2dVM1J5YVc1bktHRndjRWxrTG5SdlZYQndaWEpEWVhObEtDa3BPdzBLRFFvZ0lISmxkSFZ5YmlCYklIc2djMk5vWlcxaE9pQWlhV2RzZFRwamIyMHVZV050WlM5bWIyOHZhbk52Ym5OamFHVnRZUzh4TFRBdE1DSXNEUW9nSUNBZ0lDQWdJQ0FnSUNBZ0lDQmtZWFJoT2lCN0lHRndjRWxrVlhCd1pYSTZJR0Z3Y0Vsa1ZYQndaWElnZlEwS0lDQWdJQ0FnSUNBZ0lDQjlJRjA3RFFwOSJ9fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91c2VyX2FnZW50X3V0aWxzX2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3ciLCJuYW1lIjoidXNlcl9hZ2VudF91dGlsc19jb25maWciLCJlbmFibGVkIjp0cnVlLCJwYXJhbWV0ZXJzIjp7fX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvY2FtcGFpZ25fYXR0cmlidXRpb24vanNvbnNjaGVtYS8xLTAtMSIsImRhdGEiOnsibmFtZSI6ImNhbXBhaWduX2F0dHJpYnV0aW9uIiwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJtYXBwaW5nIjoic3RhdGljIiwiZmllbGRzIjp7Im1rdE1lZGl1bSI6WyJ1dG1fbWVkaXVtIl0sIm1rdFNvdXJjZSI6WyJ1dG1fc291cmNlIl0sIm1rdFRlcm0iOlsidXRtX3Rlcm0iXSwibWt0Q29udGVudCI6WyJ1dG1fY29udGVudCJdLCJta3RDYW1wYWlnbiI6WyJ1dG1fY2FtcGFpZ24iXX19fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cy9hcGlfcmVxdWVzdF9lbnJpY2htZW50X2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cuZW5yaWNobWVudHMiLCJuYW1lIjoiYXBpX3JlcXVlc3RfZW5yaWNobWVudF9jb25maWciLCJlbmFibGVkIjpmYWxzZSwicGFyYW1ldGVycyI6eyJpbnB1dHMiOlt7ImtleSI6InVzZXIiLCJwb2pvIjp7ImZpZWxkIjoidXNlcl9pZCJ9fSx7ImtleSI6InVzZXIiLCJqc29uIjp7ImZpZWxkIjoiY29udGV4dHMiLCJzY2hlbWFDcml0ZXJpb24iOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jbGllbnRfc2Vzc2lvbi9qc29uc2NoZW1hLzEtKi0qIiwianNvblBhdGgiOiIkLnVzZXJJZCJ9fSx7ImtleSI6ImNsaWVudCIsInBvam8iOnsiZmllbGQiOiJhcHBfaWQifX1dLCJhcGkiOnsiaHR0cCI6eyJtZXRob2QiOiJHRVQiLCJ1cmkiOiJodHRwOi8vYXBpLmFjbWUuY29tL3VzZXJzL3t7Y2xpZW50fX0ve3t1c2VyfX0/Zm9ybWF0PWpzb24iLCJ0aW1lb3V0IjoyMDAwLCJhdXRoZW50aWNhdGlvbiI6eyJodHRwQmFzaWMiOnsidXNlcm5hbWUiOiJ4eHgiLCJwYXNzd29yZCI6Inl5eSJ9fX19LCJvdXRwdXRzIjpbeyJzY2hlbWEiOiJpZ2x1OmNvbS5hY21lL3VzZXIvanNvbnNjaGVtYS8xLTAtMCIsImpzb24iOnsianNvblBhdGgiOiIkLnJlY29yZCJ9fV0sImNhY2hlIjp7InNpemUiOjMwMDAsInR0bCI6NjB9fX19XX0= --input-folder hdfs:///local/snowplow/raw-events/* --output-folder hdfs:///local/snowplow/enriched-events/ --bad-folder s3://infinigrow-logs/etlrunner/enriched/bad/run=2018-10-04-04-30-22/'
        > INFO Environment:
        >   PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/opt/aws/bin
        >   LESS_TERMCAP_md=[01;38;5;208m
        >   LESS_TERMCAP_me=[0m
        >   HISTCONTROL=ignoredups
        >   LESS_TERMCAP_mb=[01;31m
        >   AWS_AUTO_SCALING_HOME=/opt/aws/apitools/as
        >   UPSTART_JOB=rc
        >   LESS_TERMCAP_se=[0m
        >   HISTSIZE=1000
        >   HADOOP_ROOT_LOGGER=INFO,DRFA
        >   JAVA_HOME=/etc/alternatives/jre
        >   AWS_DEFAULT_REGION=us-west-2
        >   AWS_ELB_HOME=/opt/aws/apitools/elb
        >   LESS_TERMCAP_us=[04;38;5;111m
        >   EC2_HOME=/opt/aws/apitools/ec2
        >   TERM=linux
        >   XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt
        >   runlevel=3
        >   LANG=en_US.UTF-8
        >   AWS_CLOUDWATCH_HOME=/opt/aws/apitools/mon
        >   MAIL=/var/spool/mail/hadoop
        >   LESS_TERMCAP_ue=[0m
        >   LOGNAME=hadoop
        >   PWD=/
        >   LANGSH_SOURCED=1
        >   HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-14Q6W9V3K08FF/tmp
        >   _=/etc/alternatives/jre/bin/java
        >   CONSOLETYPE=serial
        >   RUNLEVEL=3
        >   LESSOPEN=||/usr/bin/lesspipe.sh %s
        >   previous=N
        >   UPSTART_EVENTS=runlevel
        >   AWS_PATH=/opt/aws
        >   USER=hadoop
        >   UPSTART_INSTANCE=
        >   PREVLEVEL=N
        >   HADOOP_LOGFILE=syslog
        >   PYTHON_INSTALL_LAYOUT=amzn
        >   HOSTNAME=ip-172-31-18-60
        >   NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat
        >   HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-14Q6W9V3K08FF
        >   EC2_AMITOOL_HOME=/opt/aws/amitools/ec2
        >   SHLVL=5
        >   HOME=/home/hadoop
        >   HADOOP_IDENT_STRING=hadoop
        > INFO redirectOutput to /mnt/var/log/hadoop/steps/s-14Q6W9V3K08FF/stdout
        > INFO redirectError to /mnt/var/log/hadoop/steps/s-14Q6W9V3K08FF/stderr
        > INFO Working dir /mnt/var/lib/hadoop/steps/s-14Q6W9V3K08FF
        > INFO ProcessRunner started child process 8876 :
        > hadoop    8876  4000  0 04:36 ?        00:00:00 bash /usr/lib/hadoop/bin/hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar spark-submit --class com.snowplowanalytics.snowplow.enrich.spark.EnrichJob --master yarn --deploy-mode cluster s3://snowplow-hosted-assets-us-west-2/3-enrich/spark-enrich/snowplow-spark-enrich-1.14.0.jar --input-format clj-tomcat --etl-timestamp 1538627422352 --iglu-config ewogICJzY2hlbWEiOiAiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3MuaWdsdS9yZXNvbHZlci1jb25maWcvanNvbnNjaGVtYS8xLTAtMSIsCiAgImRhdGEiOiB7CiAgICAiY2FjaGVTaXplIjogNTAwLAogICAgInJlcG9zaXRvcmllcyI6IFsKICAgICAgewogICAgICAgICJuYW1lIjogIklnbHUgQ2VudHJhbCIsCiAgICAgICAgInByaW9yaXR5IjogMCwKICAgICAgICAidmVuZG9yUHJlZml4ZXMiOiBbICJjb20uc25vd3Bsb3dhbmFseXRpY3MiIF0sCiAgICAgICAgImNvbm5lY3Rpb24iOiB7CiAgICAgICAgICAiaHR0cCI6IHsKICAgICAgICAgICAgInVyaSI6ICJodHRwOi8vaWdsdWNlbnRyYWwuY29tIgogICAgICAgICAgfQogICAgICAgIH0KICAgICAgfQogICAgXQogIH0KfQo= --enrichments eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9lbnJpY2htZW50cy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6W3sic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvcmVmZXJlcl9wYXJzZXIvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsibmFtZSI6InJlZmVyZXJfcGFyc2VyIiwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJpbnRlcm5hbERvbWFpbnMiOltdfX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvdWFfcGFyc2VyX2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3ciLCJuYW1lIjoidWFfcGFyc2VyX2NvbmZpZyIsImVuYWJsZWQiOnRydWUsInBhcmFtZXRlcnMiOnt9fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9hbm9uX2lwL2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7Im5hbWUiOiJhbm9uX2lwIiwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJhbm9uT2N0ZXRzIjoxfX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvaXBfbG9va3Vwcy9qc29uc2NoZW1hLzItMC0wIiwiZGF0YSI6eyJuYW1lIjoiaXBfbG9va3VwcyIsInZlbmRvciI6ImNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdyIsImVuYWJsZWQiOnRydWUsInBhcmFtZXRlcnMiOnsiZ2VvIjp7ImRhdGFiYXNlIjoiR2VvTGl0ZTItQ2l0eS5tbWRiIiwidXJpIjoiaHR0cDovL3Nub3dwbG93LWhvc3RlZC1hc3NldHMuczMuYW1hem9uYXdzLmNvbS90aGlyZC1wYXJ0eS9tYXhtaW5kIn19fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jdXJyZW5jeV9jb252ZXJzaW9uX2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJlbmFibGVkIjpmYWxzZSwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwibmFtZSI6ImN1cnJlbmN5X2NvbnZlcnNpb25fY29uZmlnIiwicGFyYW1ldGVycyI6eyJhY2NvdW50VHlwZSI6IkRFVkVMT1BFUiIsImFwaUtleSI6Int7S0VZfX0iLCJiYXNlQ3VycmVuY3kiOiJVU0QiLCJyYXRlQXQiOiJFT0RfUFJJT1IifX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cuZW5yaWNobWVudHMvc3FsX3F1ZXJ5X2VucmljaG1lbnRfY29uZmlnL2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7InZlbmRvciI6ImNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cyIsIm5hbWUiOiJzcWxfcXVlcnlfZW5yaWNobWVudF9jb25maWciLCJlbmFibGVkIjpmYWxzZSwicGFyYW1ldGVycyI6eyJpbnB1dHMiOlt7InBsYWNlaG9sZGVyIjoxLCJwb2pvIjp7ImZpZWxkIjoidXNlcl9pZCJ9fSx7InBsYWNlaG9sZGVyIjoxLCJqc29uIjp7ImZpZWxkIjoiY29udGV4dHMiLCJzY2hlbWFDcml0ZXJpb24iOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jbGllbnRfc2Vzc2lvbi9qc29uc2NoZW1hLzEtKi0qIiwianNvblBhdGgiOiIkLnVzZXJJZCJ9fSx7InBsYWNlaG9sZGVyIjoyLCJwb2pvIjp7ImZpZWxkIjoiYXBwX2lkIn19XSwiZGF0YWJhc2UiOnsicG9zdGdyZXNxbCI6eyJob3N0IjoiY2x1c3RlcjAxLnJlZHNoaWZ0LmFjbWUuY29tIiwicG9ydCI6NTQzOSwic3NsTW9kZSI6dHJ1ZSwidXNlcm5hbWUiOiJzbm93cGxvd19lbnJpY2hfcm8iLCJwYXNzd29yZCI6IjFhc0lrSmVkIiwiZGF0YWJhc2UiOiJjcm0ifX0sInF1ZXJ5Ijp7InNxbCI6IlNFTEVDVCB1c2VybmFtZSwgZW1haWxfYWRkcmVzcywgZGF0ZV9vZl9iaXJ0aCBGUk9NIHRibF91c2VycyBXSEVSRSB1c2VyID0gPyBBTkQgY2xpZW50ID0gPyBMSU1JVCAxIn0sIm91dHB1dCI6eyJleHBlY3RlZFJvd3MiOiJBVF9NT1NUX09ORSIsImpzb24iOnsic2NoZW1hIjoiaWdsdTpjb20uYWNtZS91c2VyL2pzb25zY2hlbWEvMS0wLTAiLCJkZXNjcmliZXMiOiJBTExfUk9XUyIsInByb3BlcnR5TmFtZXMiOiJDQU1FTF9DQVNFIn19LCJjYWNoZSI6eyJzaXplIjozMDAwLCJ0dGwiOjYwfX19fSx7InNjaGVtYSI6ImlnbHU6Y29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93L2V2ZW50X2ZpbmdlcnByaW50X2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3ciLCJuYW1lIjoiZXZlbnRfZmluZ2VycHJpbnRfY29uZmlnIiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJleGNsdWRlUGFyYW1ldGVycyI6WyJlaWQiLCJudWlkIiwic3RtIiwiY3YiXSwiaGFzaEFsZ29yaXRobSI6Ik1ENSJ9fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cy9odHRwX2hlYWRlcl9leHRyYWN0b3JfY29uZmlnL2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7InZlbmRvciI6ImNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cyIsIm5hbWUiOiJodHRwX2hlYWRlcl9leHRyYWN0b3JfY29uZmlnIiwiZW5hYmxlZCI6ZmFsc2UsInBhcmFtZXRlcnMiOnsiaGVhZGVyc1BhdHRlcm4iOiIuKiJ9fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cy93ZWF0aGVyX2VucmljaG1lbnRfY29uZmlnL2pzb25zY2hlbWEvMS0wLTAiLCJkYXRhIjp7ImVuYWJsZWQiOmZhbHNlLCJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cuZW5yaWNobWVudHMiLCJuYW1lIjoid2VhdGhlcl9lbnJpY2htZW50X2NvbmZpZyIsInBhcmFtZXRlcnMiOnsiYXBpS2V5Ijoie3tLRVl9fSIsImNhY2hlU2l6ZSI6NTEwMCwiZ2VvUHJlY2lzaW9uIjoxLCJhcGlIb3N0IjoiaGlzdG9yeS5vcGVud2VhdGhlcm1hcC5vcmciLCJ0aW1lb3V0Ijo1fX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvY29va2llX2V4dHJhY3Rvcl9jb25maWcvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsibmFtZSI6ImNvb2tpZV9leHRyYWN0b3JfY29uZmlnIiwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJjb29raWVzIjpbInNwIl19fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9qYXZhc2NyaXB0X3NjcmlwdF9jb25maWcvanNvbnNjaGVtYS8xLTAtMCIsImRhdGEiOnsidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwibmFtZSI6ImphdmFzY3JpcHRfc2NyaXB0X2NvbmZpZyIsImVuYWJsZWQiOmZhbHNlLCJwYXJhbWV0ZXJzIjp7InNjcmlwdCI6IlpuVnVZM1JwYjI0Z2NISnZZMlZ6Y3lobGRtVnVkQ2tnZXcwS0RRb2dJSFpoY2lCd2JHRjBabTl5YlNBOUlHVjJaVzUwTG1kbGRGQnNZWFJtYjNKdEtDa3NEUW9nSUNBZ0lDQmhjSEJKWkNBZ0lDQTlJR1YyWlc1MExtZGxkRUZ3Y0Y5cFpDZ3BPdzBLRFFvZ0lHbG1JQ2h3YkdGMFptOXliU0E5UFNBaWMyVnlkbVZ5SWlBbUppQmhjSEJKWkNBaFBTQWljMlZqY21WMElpa2dldzBLSUNBZ0lIUm9jbTkzSUNKVFpYSjJaWEl0YzJsa1pTQmxkbVZ1ZENCb1lYTWdhVzUyWVd4cFpDQmhjSEJmYVdRNklDSWdLeUJoY0hCSlpEc05DaUFnZlEwS0lDQU5DaUFnYVdZZ0tHRndjRWxrSUQwOUlHNTFiR3dwSUhzTkNpQWdJQ0J5WlhSMWNtNGdXMTA3RFFvZ0lIME5DZzBLSUNBdkx5QlZjMlVnYm1WM0lGTjBjbWx1WnlncElHSmxZMkYxYzJVZ2FIUjBjRG92TDI1bGJITnZibmRsYkd4ekxtNWxkQzh5TURFeUx6QXlMMnB6YjI0dGMzUnlhVzVuYVdaNUxYZHBkR2d0YldGd2NHVmtMWFpoY21saFlteGxjeThOQ2lBZ2RtRnlJR0Z3Y0Vsa1ZYQndaWElnUFNCdVpYY2dVM1J5YVc1bktHRndjRWxrTG5SdlZYQndaWEpEWVhObEtDa3BPdzBLRFFvZ0lISmxkSFZ5YmlCYklIc2djMk5vWlcxaE9pQWlhV2RzZFRwamIyMHVZV050WlM5bWIyOHZhbk52Ym5OamFHVnRZUzh4TFRBdE1DSXNEUW9nSUNBZ0lDQWdJQ0FnSUNBZ0lDQmtZWFJoT2lCN0lHRndjRWxrVlhCd1pYSTZJR0Z3Y0Vsa1ZYQndaWElnZlEwS0lDQWdJQ0FnSUNBZ0lDQjlJRjA3RFFwOSJ9fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91c2VyX2FnZW50X3V0aWxzX2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3ciLCJuYW1lIjoidXNlcl9hZ2VudF91dGlsc19jb25maWciLCJlbmFibGVkIjp0cnVlLCJwYXJhbWV0ZXJzIjp7fX19LHsic2NoZW1hIjoiaWdsdTpjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cvY2FtcGFpZ25fYXR0cmlidXRpb24vanNvbnNjaGVtYS8xLTAtMSIsImRhdGEiOnsibmFtZSI6ImNhbXBhaWduX2F0dHJpYnV0aW9uIiwidmVuZG9yIjoiY29tLnNub3dwbG93YW5hbHl0aWNzLnNub3dwbG93IiwiZW5hYmxlZCI6dHJ1ZSwicGFyYW1ldGVycyI6eyJtYXBwaW5nIjoic3RhdGljIiwiZmllbGRzIjp7Im1rdE1lZGl1bSI6WyJ1dG1fbWVkaXVtIl0sIm1rdFNvdXJjZSI6WyJ1dG1fc291cmNlIl0sIm1rdFRlcm0iOlsidXRtX3Rlcm0iXSwibWt0Q29udGVudCI6WyJ1dG1fY29udGVudCJdLCJta3RDYW1wYWlnbiI6WyJ1dG1fY2FtcGFpZ24iXX19fX0seyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy5lbnJpY2htZW50cy9hcGlfcmVxdWVzdF9lbnJpY2htZW50X2NvbmZpZy9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJ2ZW5kb3IiOiJjb20uc25vd3Bsb3dhbmFseXRpY3Muc25vd3Bsb3cuZW5yaWNobWVudHMiLCJuYW1lIjoiYXBpX3JlcXVlc3RfZW5yaWNobWVudF9jb25maWciLCJlbmFibGVkIjpmYWxzZSwicGFyYW1ldGVycyI6eyJpbnB1dHMiOlt7ImtleSI6InVzZXIiLCJwb2pvIjp7ImZpZWxkIjoidXNlcl9pZCJ9fSx7ImtleSI6InVzZXIiLCJqc29uIjp7ImZpZWxkIjoiY29udGV4dHMiLCJzY2hlbWFDcml0ZXJpb24iOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jbGllbnRfc2Vzc2lvbi9qc29uc2NoZW1hLzEtKi0qIiwianNvblBhdGgiOiIkLnVzZXJJZCJ9fSx7ImtleSI6ImNsaWVudCIsInBvam8iOnsiZmllbGQiOiJhcHBfaWQifX1dLCJhcGkiOnsiaHR0cCI6eyJtZXRob2QiOiJHRVQiLCJ1cmkiOiJodHRwOi8vYXBpLmFjbWUuY29tL3VzZXJzL3t7Y2xpZW50fX0ve3t1c2VyfX0/Zm9ybWF0PWpzb24iLCJ0aW1lb3V0IjoyMDAwLCJhdXRoZW50aWNhdGlvbiI6eyJodHRwQmFzaWMiOnsidXNlcm5hbWUiOiJ4eHgiLCJwYXNzd29yZCI6Inl5eSJ9fX19LCJvdXRwdXRzIjpbeyJzY2hlbWEiOiJpZ2x1OmNvbS5hY21lL3VzZXIvanNvbnNjaGVtYS8xLTAtMCIsImpzb24iOnsianNvblBhdGgiOiIkLnJlY29yZCJ9fV0sImNhY2hlIjp7InNpemUiOjMwMDAsInR0bCI6NjB9fX19XX0= --input-folder hdfs:///local/snowplow/raw-events/* --output-folder hdfs:///local/snowplow/enriched-events/ --bad-folder s3://infinigrow-logs/etlrunner/enriched/bad/run=2018-10-04-04-30-22/
        > 2018-10-04T04:36:30.087Z INFO HadoopJarStepRunner.Runner: startRun() called for s-14Q6W9V3K08FF Child Pid: 8876
        > INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
        > INFO Process still running
        > INFO Process still running
        > INFO Process still running
        > INFO Process still running
        > INFO Process still running
        > ...
        > ...
        > ...

strerr:

    Warning: Skip remote jar s3://snowplow-hosted-assets-us-west-2/3-enrich/spark-enrich/snowplow-spark-enrich-1.14.0.jar.
    18/10/04 04:36:34 INFO RMProxy: Connecting to ResourceManager at ip-172-31-18-60.us-west-2.compute.internal/172.31.18.60:8032
    18/10/04 04:36:34 INFO Client: Requesting a new application from cluster with 2 NodeManagers
    18/10/04 04:36:34 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (6144 MB per container)
    18/10/04 04:36:34 INFO Client: Will allocate AM container, with 6143 MB memory including 558 MB overhead
    18/10/04 04:36:34 INFO Client: Setting up container launch context for our AM
    18/10/04 04:36:34 INFO Client: Setting up the launch environment for our AM container
    18/10/04 04:36:34 INFO Client: Preparing resources for our AM container
    18/10/04 04:36:36 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
    18/10/04 04:36:38 INFO Client: Uploading resource file:/mnt/tmp/spark-778585da-c362-4270-bab7-4598895a7872/__spark_libs__7664507486617369497.zip -> hdfs://ip-172-31-18-60.us-west-2.compute.internal:8020/user/hadoop/.sparkStaging/application_1538627609524_0003/__spark_libs__7664507486617369497.zip
    18/10/04 04:36:41 WARN RoleMappings: Found no mappings configured with 'fs.s3.authorization.roleMapping', credentials resolution may not work as expected
    18/10/04 04:36:41 INFO Client: Uploading resource s3://snowplow-hosted-assets-us-west-2/3-enrich/spark-enrich/snowplow-spark-enrich-1.14.0.jar -> hdfs://ip-172-31-18-60.us-west-2.compute.internal:8020/user/hadoop/.sparkStaging/application_1538627609524_0003/snowplow-spark-enrich-1.14.0.jar
    18/10/04 04:36:41 INFO S3NativeFileSystem: Opening 's3://snowplow-hosted-assets-us-west-2/3-enrich/spark-enrich/snowplow-spark-enrich-1.14.0.jar' for reading
    18/10/04 04:36:43 INFO Client: Uploading resource file:/mnt/tmp/spark-778585da-c362-4270-bab7-4598895a7872/__spark_conf__1278190290174767064.zip -> hdfs://ip-172-31-18-60.us-west-2.compute.internal:8020/user/hadoop/.sparkStaging/application_1538627609524_0003/__spark_conf__.zip
    18/10/04 04:36:43 INFO SecurityManager: Changing view acls to: hadoop
    18/10/04 04:36:43 INFO SecurityManager: Changing modify acls to: hadoop
    18/10/04 04:36:43 INFO SecurityManager: Changing view acls groups to: 
    18/10/04 04:36:43 INFO SecurityManager: Changing modify acls groups to: 
    18/10/04 04:36:43 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop); groups with view permissions: Set(); users  with modify permissions: Set(hadoop); groups with modify permissions: Set()
    18/10/04 04:36:43 INFO Client: Submitting application application_1538627609524_0003 to ResourceManager
    18/10/04 04:36:43 INFO YarnClientImpl: Submitted application application_1538627609524_0003
    18/10/04 04:36:44 INFO Client: Application report for application_1538627609524_0003 (state: ACCEPTED)
    18/10/04 04:36:44 INFO Client: 
    	 client token: N/A
    	 diagnostics: N/A
    	 ApplicationMaster host: N/A
    	 ApplicationMaster RPC port: -1
    	 queue: default
    	 start time: 1538627803560
    	 final status: UNDEFINED
    	 tracking URL: http://ip-172-31-18-60.us-west-2.compute.internal:20888/proxy/application_1538627609524_0003/
    	 user: hadoop
    18/10/04 04:36:45 INFO Client: Application report for application_1538627609524_0003 (state: ACCEPTED)
    18/10/04 04:36:46 INFO Client: Application report for application_1538627609524_0003 (state: ACCEPTED)
    18/10/04 04:36:47 INFO Client: Application report for application_1538627609524_0003 (state: ACCEPTED)
    18/10/04 04:36:48 INFO Client: Application report for application_1538627609524_0003 (state: ACCEPTED)
    18/10/04 04:36:49 INFO Client: Application report for application_1538627609524_0003 (state: ACCEPTED)
    18/10/04 04:36:50 INFO Client: Application report for application_1538627609524_0003 (state: ACCEPTED)
    18/10/04 04:36:51 INFO Client: Application report for application_1538627609524_0003 (state: ACCEPTED)
    18/10/04 04:36:52 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
    18/10/04 04:36:52 INFO Client: 
    	 client token: N/A
    	 diagnostics: N/A
    	 ApplicationMaster host: 172.31.28.194
    	 ApplicationMaster RPC port: 0
    	 queue: default
    	 start time: 1538627803560
    	 final status: UNDEFINED
    	 tracking URL: http://ip-172-31-18-60.us-west-2.compute.internal:20888/proxy/application_1538627609524_0003/
    	 user: hadoop
    18/10/04 04:36:53 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
    18/10/04 04:36:54 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
    18/10/04 04:36:55 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
    18/10/04 04:36:56 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
    18/10/04 04:36:57 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
    18/10/04 04:36:58 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
    18/10/04 04:36:59 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
    18/10/04 04:37:00 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)
    18/10/04 04:37:01 INFO Client: Application report for application_1538627609524_0003 (state: RUNNING)

log from spark console:

    org.apache.spark.sql.DataFrameWriter.text(DataFrameWriter.scala:555)
    com.snowplowanalytics.snowplow.enrich.spark.EnrichJob.run(EnrichJob.scala:212)
    com.snowplowanalytics.snowplow.enrich.spark.EnrichJob$.run(EnrichJob.scala:88)
    com.snowplowanalytics.snowplow.enrich.spark.SparkJob$class.main(SparkJob.scala:33)
    com.snowplowanalytics.snowplow.enrich.spark.EnrichJob$.main(EnrichJob.scala:52)
    com.snowplowanalytics.snowplow.enrich.spark.EnrichJob.main(EnrichJob.scala)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:498)
    org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:635)

What can cause this? Also is it risky to cancel the cluster and run it again? Might cause losing data?


#2

Hi @ocherfas,

There’s no risk to terminate it at enrich step and no data will be lost. However, there are some moments which you need to consider (see explanation here).

  • What version of EER are you using?
  • How many files do you have in the processing bucket and what is their size?
  • What EMR cluster configuration are you using (master and core nodes)?