Dataflow runner EMR instances automatically deleted

For engineers AWS batch pipeline (Legacy)
1

Hi, everyone.

I found these events in AWS Cloudtrail, where the instances used for the Data-flow runner (EMR) in charge of enrichment and shredding were deleted/terminated. It seems like the source was the EMR itself since the implicated user agent was "userAgent": "elasticmapreduce.amazonaws.com", but I couldn’t find out more about the root cause, as this was not a requested action. Can you guys help me finding some additional clues about that? Here is the detailed event in Cloudtrail:

Thank you!

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROA36KBMHJUXPNMMUMJA:CCSSession",
        "arn": "arn:aws:sts::---:assumed-role/EMR_DefaultRole/CCSSession",
        "accountId": "---",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "AROA36KBMHJUXPNMMUMJA",
                "arn": "arn:aws:iam::---:role/EMR_DefaultRole",
                "accountId": "---",
                "userName": "EMR_DefaultRole"
            },
            "webIdFederationData": {},
            "attributes": {
                "creationDate": "2022-02-22T11:51:30Z",
                "mfaAuthenticated": "false"
            }
        },
        "invokedBy": "elasticmapreduce.amazonaws.com"
    },
    "eventTime": "2022-02-22T11:51:30Z",
    "eventSource": "ec2.amazonaws.com",
    "eventName": "TerminateInstances",
    "awsRegion": "us-west-2",
    "sourceIPAddress": "elasticmapreduce.amazonaws.com",
    "userAgent": "elasticmapreduce.amazonaws.com",
    "requestParameters": {
        "instancesSet": {
            "items": [
                {
                    "instanceId": "i-04d95ecd582ec353b"
                }
            ]
        }
    },
    "responseElements": {
        "requestId": "412c3afa-9f23-4932-8a06-181f803101a5",
        "instancesSet": {
            "items": [
                {
                    "instanceId": "i-04d95ecd582ec353b",
                    "currentState": {
                        "code": 32,
                        "name": "shutting-down"
                    },
                    "previousState": {
                        "code": 16,
                        "name": "running"
                    }
                }
            ]
        }
    },
    "requestID": "412c3afa-9f23-4932-8a06-181f803101a5",
    "eventID": "fe6cf835-1e97-48ac-a338-3d6683d707a1",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "eventCategory": "Management"
}
2

I’m not sure that the termination reason is available via CloudTrail but it will be available via either the EMR API or in the interface itself

https://us-west-2.console.aws.amazon.com/elasticmapreduce/home?region=us-west-2#cluster-details:j-YOURJOBIDHERE (replacing your job id) and the final event should be a termination event that specifies the type of termination.

Older jobs have a tendency to not always show up in the UI so the equivalent in the AWS CLI is aws emr describe-cluster --cluster-id j-JOBID --query Cluster.Status