Hi team,
My storage loader job failed yesterday and today after successful completion of EMR ETL job.
But ran successfully when I reran it. I have also contacted AWS for the same issue and they said there were reboots in my cluster around the time. Error logs says(both day I have same error):
Starting EmrEtlRunner
D, [2016-11-21T02:45:21.524000 #7169] DEBUG -- : Staging raw logs...
moving files from s3://udmd-global-p-raw-logs/ to s3://udmd-p-etl/processing/
(t6) MOVE udmd-global-p-raw-logs/i-040f619e8e8f21d11/_var_log_tomcat8_rotated_localhost_access_log.txt1479628862.gz -> udmd-p-etl/processing/var_log_tomcat8_rotated_localhost_access_log.2016-11-20-08.eu-west-1.i-040f619e8e8f21d11.txt.gz(t4) MOVE udmd-global-p-raw-logs/i-040f619e8e8f21d11/_var_log_tomcat8_rotated_localhost_access_log.txt1479621662.gz -> udmd-p-etl/processing/var_log_tomcat8_rotated_localhost_access_log.2016-11-20-06.eu-west-1.i-040f619e8e8f21d11.txt.gz
........
........
D, [2016-11-21T02:45:58.980000 #7169] DEBUG -- : Waiting a minute to allow S3 to settle (eventual consistency)
D, [2016-11-21T02:46:58.986000 #7169] DEBUG -- : Initializing EMR jobflow
D, [2016-11-21T02:47:01.090000 #7169] DEBUG -- : EMR jobflow j-ZM1AGW29EOJU started, waiting for jobflow to complete...
D, [2016-11-21T05:23:19.099000 #7169] DEBUG -- : EMR jobflow j-ZM1AGW29EOJU completed successfully.
D, [2016-11-21T05:23:19.100000 #7169] DEBUG -- : Archiving CloudFront logs...
moving files from s3://udmd-p-etl/processing/ to s3://udmd-p-archive/raw/
......
......
I, [2016-11-21T05:23:44.263000 #7169] INFO -- : Completed successfully
Starting Storage Loader
Loading Snowplow events and shredded types into My Redshift database (Redshift cluster)...
**Unexpected error: Java::Default::PSQLException error executing ANALYZE statements**: BEGIN;
**ANALYZE landing.events**;
ANALYZE landing.com_google_analytics_enhanced_ecommerce_action_1;
ANALYZE landing.com_google_analytics_enhanced_ecommerce_action_field_object_1;
ANALYZE landing.com_google_analytics_enhanced_ecommerce_impression_field_object_1;
ANALYZE landing.com_google_analytics_enhanced_ecommerce_product_field_object_1;
ANALYZE landing.com_snowplowanalytics_snowplow_add_to_cart_1;
ANALYZE landing.com_snowplowanalytics_snowplow_remove_from_cart_1;
ANALYZE landing.com_snowplowanalytics_snowplow_ua_parser_context_1;
ANALYZE landing.udmd_unileversolutions_componentvideo_1;
ANALYZE landing.udmd_unileversolutions_custom_data_1;
ANALYZE landing.udmd_unileversolutions_page_1;
ANALYZE landing.udmd_unileversolutions_product_1;
ANALYZE landing.udmd_unileversolutions_promotion_1;
COMMIT;: **An I/O error occurred while sending to the backend.**
/data/apps/SnowplowRealeases/r75/snowplow-master/4-storage/storage-loader/deploy/snowplow-storage-loader!/storage-loader/lib/snowplow-storage-loader/redshift_loader.rb:111:in `load_events_and_shredded_types'
file:/data/apps/SnowplowRealeases/r75/snowplow-master/4-storage/storage-loader/deploy/snowplow-storage-loader!/storage-loader/bin/snowplow-storage-loader:54:in `(root)'
org/jruby/RubyArray.java:1613:in `each'
file:/data/apps/SnowplowRealeases/r75/snowplow-master/4-storage/storage-loader/deploy/snowplow-storage-loader!/storage-loader/bin/snowplow-storage-loader:51:in `(root)'
org/jruby/RubyKernel.java:1091:in `load'
file:/data/apps/SnowplowRealeases/r75/snowplow-master/4-storage/storage-loader/deploy/snowplow-storage-loader!/META-INF/main.rb:1:in `(root)'
org/jruby/RubyKernel.java:1072:in `require'
file:/data/apps/SnowplowRealeases/r75/snowplow-master/4-storage/storage-loader/deploy/snowplow-storage-loader!/META-INF/main.rb:1:in `(root)'
/tmp/jruby426935701937160700extract/jruby-stdlib-1.7.20.1.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1:in `(root)'
error in running the Storage Loader, exiting with return code1.
Can you please help me with this and let me know how I can fix this.
Many Thanks!
Deepak Bhatt