Further the following container log on the EMR CORE instance has the last line perpetually repeated:
2017-06-20 16:06:34,895 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1497974621025_0001_000001
2017-06-20 16:06:35,464 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2017-06-20 16:06:35,464 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 1 cluster_timestamp: 1497974621025 } attemptId: 1 } keyId: -135228129)
2017-06-20 16:06:35,698 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2017-06-20 16:06:35,699 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config org.apache.hadoop.mapred.DirectFileOutputCommitter
2017-06-20 16:06:35,740 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2017-06-20 16:06:36,252 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.DirectFileOutputCommitter
2017-06-20 16:06:36,382 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2017-06-20 16:06:36,383 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2017-06-20 16:06:36,384 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2017-06-20 16:06:36,385 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2017-06-20 16:06:36,385 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2017-06-20 16:06:36,390 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2017-06-20 16:06:36,391 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2017-06-20 16:06:36,392 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2017-06-20 16:06:36,428 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://ip-172-30-1-230.ec2.internal:8020]
2017-06-20 16:06:36,452 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://ip-172-30-1-230.ec2.internal:8020]
2017-06-20 16:06:36,482 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://ip-172-30-1-230.ec2.internal:8020]
2017-06-20 16:06:36,503 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Perms after creating 488, Expected: 504
2017-06-20 16:06:36,503 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Explicitly setting permissions to : 504, rwxrwx—
2017-06-20 16:06:36,508 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
2017-06-20 16:06:36,552 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2017-06-20 16:06:36,635 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2017-06-20 16:06:36,641 INFO [main] com.amazon.ws.emr.hadoop.metrics2.sink.cloudwatch.CloudWatchSink: Initializing the CloudWatchSink for metrics.
2017-06-20 16:06:36,699 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: Sink cloudwatch started
2017-06-20 16:06:36,772 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 300 second(s).
2017-06-20 16:06:36,772 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
2017-06-20 16:06:36,780 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1497974621025_0001 to jobTokenSecretManager
2017-06-20 16:06:36,876 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1497974621025_0001 because: not enabled; too many reduces;
2017-06-20 16:06:36,887 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1497974621025_0001 = 36350. Number of splits = 1
2017-06-20 16:06:36,889 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1497974621025_0001 = 10
2017-06-20 16:06:36,889 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1497974621025_0001Job Transitioned from NEW to INITED
2017-06-20 16:06:36,890 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1497974621025_0001.
2017-06-20 16:06:36,911 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2017-06-20 16:06:36,916 INFO [Socket Reader #1 for port 35132] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 35132
2017-06-20 16:06:36,932 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2017-06-20 16:06:36,937 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2017-06-20 16:06:36,938 INFO [IPC Server listener on 35132] org.apache.hadoop.ipc.Server: IPC Server listener on 35132: starting
2017-06-20 16:06:36,938 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at ip-172-30-4-165.ec2.internal/172.30.4.165:35132
2017-06-20 16:06:37,009 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2017-06-20 16:06:37,016 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2017-06-20 16:06:37,020 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
2017-06-20 16:06:37,025 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter ‘safety’ (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2017-06-20 16:06:37,028 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
2017-06-20 16:06:37,029 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
2017-06-20 16:06:37,031 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
2017-06-20 16:06:37,031 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2017-06-20 16:06:37,387 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
2017-06-20 16:06:37,388 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 39405
2017-06-20 16:06:37,389 INFO [main] org.mortbay.log: jetty-6.1.26-emr
2017-06-20 16:06:37,416 INFO [main] org.mortbay.log: Extract jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.7.3-amzn-2.jar!/webapps/mapreduce to /mnt/yarn/usercache/hadoop/appcache/application_1497974621025_0001/container_1497974621025_0001_01_000001/tmp/Jetty_0_0_0_0_39405_mapreduce____.namnp6/webapp
2017-06-20 16:06:38,560 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:39405
2017-06-20 16:06:38,562 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app mapreduce started at 39405
2017-06-20 16:06:38,565 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1497974621025_0001
2017-06-20 16:06:38,568 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2017-06-20 16:06:38,570 INFO [Socket Reader #1 for port 34775] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 34775
2017-06-20 16:06:38,574 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2017-06-20 16:06:38,574 INFO [IPC Server listener on 34775] org.apache.hadoop.ipc.Server: IPC Server listener on 34775: starting
2017-06-20 16:06:38,607 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
2017-06-20 16:06:38,607 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
2017-06-20 16:06:38,607 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
2017-06-20 16:06:38,664 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at ip-172-30-1-230.ec2.internal/172.30.1.230:8030
2017-06-20 16:06:38,741 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: maxContainerCapability: <memory:1792, vCores:128>
2017-06-20 16:06:38,741 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: queue: default
2017-06-20 16:06:38,745 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
2017-06-20 16:06:38,745 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10
2017-06-20 16:06:38,747 INFO [main] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
2017-06-20 16:06:38,753 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1497974621025_0001Job Transitioned from INITED to SETUP
2017-06-20 16:06:38,757 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
2017-06-20 16:06:38,764 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1497974621025_0001Job Transitioned from SETUP to RUNNING
2017-06-20 16:06:38,780 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved ip-172-30-4-165.ec2.internal to /default-rack
2017-06-20 16:06:38,787 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1497974621025_0001_m_000000 Task Transitioned from NEW to SCHEDULED
2017-06-20 16:06:38,787 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1497974621025_0001_r_000000 Task Transitioned from NEW to SCHEDULED
2017-06-20 16:06:38,787 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1497974621025_0001_r_000001 Task Transitioned from NEW to SCHEDULED
2017-06-20 16:06:38,788 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1497974621025_0001_r_000002 Task Transitioned from NEW to SCHEDULED
2017-06-20 16:06:38,788 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1497974621025_0001_r_000003 Task Transitioned from NEW to SCHEDULED
2017-06-20 16:06:38,788 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1497974621025_0001_r_000004 Task Transitioned from NEW to SCHEDULED
2017-06-20 16:06:38,788 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1497974621025_0001_r_000005 Task Transitioned from NEW to SCHEDULED
2017-06-20 16:06:38,789 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1497974621025_0001_r_000006 Task Transitioned from NEW to SCHEDULED
2017-06-20 16:06:38,789 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1497974621025_0001_r_000007 Task Transitioned from NEW to SCHEDULED
2017-06-20 16:06:38,789 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1497974621025_0001_r_000008 Task Transitioned from NEW to SCHEDULED
2017-06-20 16:06:38,789 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1497974621025_0001_r_000009 Task Transitioned from NEW to SCHEDULED
2017-06-20 16:06:38,790 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1497974621025_0001_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2017-06-20 16:06:38,791 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1497974621025_0001_r_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2017-06-20 16:06:38,791 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1497974621025_0001_r_000001_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2017-06-20 16:06:38,791 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1497974621025_0001_r_000002_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2017-06-20 16:06:38,791 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1497974621025_0001_r_000003_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2017-06-20 16:06:38,791 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1497974621025_0001_r_000004_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2017-06-20 16:06:38,791 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1497974621025_0001_r_000005_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2017-06-20 16:06:38,791 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1497974621025_0001_r_000006_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2017-06-20 16:06:38,791 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1497974621025_0001_r_000007_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2017-06-20 16:06:38,792 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1497974621025_0001_r_000008_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2017-06-20 16:06:38,792 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1497974621025_0001_r_000009_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2017-06-20 16:06:38,794 INFO [Thread-83] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:896, vCores:1>
2017-06-20 16:06:38,805 INFO [Thread-83] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: reduceResourceRequest:<memory:1792, vCores:1>
2017-06-20 16:06:38,838 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1497974621025_0001, File: hdfs://ip-172-30-1-230.ec2.internal:8020/tmp/hadoop-yarn/staging/hadoop/.staging/job_1497974621025_0001/job_1497974621025_0001_1.jhist
2017-06-20 16:06:39,748 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:10 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
2017-06-20 16:06:39,784 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1497974621025_0001: ask=3 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:0, vCores:0> knownNMs=1
2017-06-20 16:06:39,785 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps
2017-06-20 16:06:40,789 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps