ArgumentError (AWS EMR API Error (ExpiredTokenException): The security token included in the request is expired)


#1

we are running AWS EMR batch using jenkins and for long running EMR (Say more than 6 hours or so) we are seeing ExpiredToken exception in jenkins logs. But the EMR Cluster keeps running behind .

Looks like there is some timeout config setup between AWS EMR and config.yml which throws this exception in the batch processor.

Run the ETL job on EMR 03:49:08 ./snowplow-emr-etl-runner --config {RUNNER_CONFIG} --resolver {RUNNER_RESOLVER} --enrichments ${RUNNER_ENRICHMENTS} --skip


ArgumentError (AWS EMR API Error (ExpiredTokenException): The security token included in the request is expired)


Exception below

[SSH] executing…

[] DEBUG – : Initializing EMR jobflow 03:49:54 D,DEBUG – : EMR jobflow j-XXXXXXX started, waiting for jobflow to complete… 09:45:01 F,
FATAL – : 09:45:01 09:45:01 ArgumentError (AWS EMR API Error (ExpiredTokenException): The security token included in the request is expired): 09:45:01 /home/ec2-user/snowplow-emr-etl-runner!/gems/elasticity-6.0.8/lib/elasticity/aws_session.rb:33:in submit' **09:45:01** /home/ec2-user/snowplow-emr-etl-runner!/gems/elasticity-6.0.8/lib/elasticity/aws_session.rb:29:insubmit’ 09:45:01 /home/ec2-user/snowplow-emr-etl-runner!/gems/elasticity-6.0.8/lib/elasticity/emr.rb:206:in list_steps' **09:45:01** /home/ec2-user/snowplow-emr-etl-runner!/gems/elasticity-6.0.8/lib/elasticity/job_flow.rb:174:incluster_step_status’ 09:45:01 /home/ec2-user/snowplow-emr-etl-runner!/emr-etl-runner/lib/snowplow-emr-etl-runner/emr_job.rb:538:in wait_for' **09:45:01** /home/ec2-user/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts/method_reference.rb:46:insend_to’ 09:45:01 /home/ec2-user/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts.rb:305:in call_with' **09:45:01** /home/ec2-user/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts/decorators.rb:159:incommon_method_added’ 09:45:01 /home/ec2-user/snowplow-emr-etl-runner!/emr-etl-runner/lib/snowplow-emr-etl-runner/emr_job.rb:466:in run' **09:45:01** /home/ec2-user/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts/method_reference.rb:46:insend_to’ 09:45:01 /home/ec2-user/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts.rb:305:in call_with' **09:45:01** /home/ec2-user/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts/decorators.rb:159:incommon_method_added’ 09:45:01 /home/ec2-user/snowplow-emr-etl-runner!/emr-etl-runner/lib/snowplow-emr-etl-runner/runner.rb:68:in run' **09:45:01** /home/ec2-user/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts/method_reference.rb:46:insend_to’ 09:45:01 /home/ec2-user/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts.rb:305:in call_with' **09:45:01** /home/ec2-user/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts/decorators.rb:159:incommon_method_added’ 09:45:01 file:/home/ec2-user/snowplow-emr-etl-runner!/emr-etl-runner/bin/snowplow-emr-etl-runner:39:in (root)' **09:45:01** org/jruby/RubyKernel.java:1059:inload’ 09:45:01 file:/home/ec2-user/snowplow-emr-etl-runner!/META-INF/main.rb:1:in (root)' **09:45:01** org/jruby/RubyKernel.java:1040:inrequire’ 09:45:01 file:/home/ec2-user/snowplow-emr-etl-runner!/META-INF/main.rb:1:in (root)' **09:45:01** /tmp/jruby3012283755745909084extract/jruby-stdlib-1.7.24.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1:in(root)’ 09:45:01 09:45:01

is there a configuration that we can fix this in the batch processor ?


#2

Are you using STS / a role that generates STS? If so what is the maximum API session duration set to?

Generally you want to have Spark Enrich run faster than 6 hours so a solution to this could just be to provision more instances in the EMR job (or instances with more memory / compute).