Socket Errors during archiving process

For engineers Troubleshooting
1

During both the ‘Staging raw logs…’ and ‘Archiving Snowplow events…’ stages, Sluice continually has problems moving files, e.g.

Problem copying bucket/enriched/good/run=YYYY-MM-DD-h-m-s/part-00001. Retrying

Typically, this will result in the following error being thrown before the archiving process has completed:

Unexpected error: Socket closed (OpenSSL::SSL::SSLError)
org/jruby/ext/openssl/SSLSocket.java:220:in `connect_nonblock'
uri:classloader:/gems/excon-0.51.0/lib/excon/ssl_socket.rb:120:in `initialize'
uri:classloader:/gems/excon-0.51.0/lib/excon/connection.rb:404:in `socket'
uri:class`loader:/gems/excon-0.51.0/lib/excon/connection.rb:106:in `request_call'
uri:classloader:/gems/excon-0.51.0/lib/excon/middlewares/mock.rb:47:in `request_call'
uri:classloader:/gems/excon-0.51.0/lib/excon/middlewares/instrumentor.rb:25:in `request_call'
uri:classloader:/gems/excon-0.51.0/lib/excon/middlewares/base.rb:15:in `request_call'
uri:classloader:/gems/excon-0.51.0/lib/excon/middlewares/base.rb:15:in `request_call'
uri:classloader:/gems/excon-0.51.0/lib/excon/middlewares/base.rb:15:in `request_call'
uri:classloader:/gems/excon-0.51.0/lib/excon/connection.rb:250:in `request'
uri:classloader:/gems/fog-xml-0.1.2/lib/fog/xml/sax_parser_connection.rb:35:in `request'
uri:classloader:/gems/fog-xml-0.1.2/lib/fog/xml/connection.rb:7:in `request'
uri:classloader:/gems/fog-1.25.0/lib/fog/aws/storage.rb:521:in `_request'
uri:classloader:/gems/fog-1.25.0/lib/fog/aws/storage.rb:516:in `request'
uri:classloader:/gems/fog-1.25.0/lib/fog/aws/requests/storage/copy_object.rb:32:in `copy_object'
uri:classloader:/gems/fog-1.25.0/lib/fog/aws/models/storage/file.rb:92:in `copy'
uri:classloader:/gems/sluice-0.3.4/lib/sluice/storage/s3/s3.rb:623:in `block in retry_x'
org/jruby/ext/timeout/Timeout.java:117:in `timeout'
uri:classloader:/gems/sluice-0.3.4/lib/sluice/storage/s3/s3.rb:622:in `retry_x'
uri:classloader:/gems/sluice-0.3.4/lib/sluice/storage/s3/s3.rb:548:in `block in process_files'
org/jruby/RubyKernel.java:1290:in `loop'
uri:classloader:/gems/sluice-0.3.4/lib/sluice/storage/s3/s3.rb:412:in `block in process_files'

Has any body encountered/had success in solving a similar error?

2

Hi @samf89 - we are planning on sunsetting Sluice because file moves should be done from the EMR cluster itself. We see the failures you mention but they are relatively infrequent across our customer base - for them to be happening continually for a single pipeline is unusual.

Are you perhaps running ErmEtlRunner/StorageLoader somewhere other than EC2, or on an instance type with very low networking capability?

3

Hi, thanks for the reply.

The EtlRunner/StorageLoader is currently being ran on an m1.small EC2 instance type.

4

Right - I’d try it on a larger instance type and see if the problem goes away…

5

@alex We are facing similar issue but with the EMR job. Our EMR job is not even getting started due to OpenSSL error.

D, [2019-11-17T01:00:13.794000 #14479] DEBUG – : Initializing EMR jobflow
Excon::Error::Socket: Socket closed (OpenSSL::SSL::SSLError)
raise_socket_error at uri:classloader:/gems/excon-0.52.0/lib/excon/connection.rb:421
request_call at uri:classloader:/gems/excon-0.52.0/lib/excon/connection.rb:186
request_call at uri:classloader:/gems/excon-0.52.0/lib/excon/middlewares/mock.rb:48
request_call at uri:classloader:/gems/excon-0.52.0/lib/excon/middlewares/instrumentor.rb:26
request_call at uri:classloader:/gems/excon-0.52.0/lib/excon/middlewares/base.rb:16
request_call at uri:classloader:/gems/excon-0.52.0/lib/excon/middlewares/base.rb:16
request_call at uri:classloader:/gems/excon-0.52.0/lib/excon/middlewares/base.rb:16
request at uri:classloader:/gems/excon-0.52.0/lib/excon/connection.rb:249
error_call at uri:classloader:/gems/excon-0.52.0/lib/excon/middlewares/idempotent.rb:27
error_call at uri:classloader:/gems/excon-0.52.0/lib/excon/middlewares/base.rb:11
error_call at uri:classloader:/gems/excon-0.52.0/lib/excon/middlewares/base.rb:11
request at uri:classloader:/gems/excon-0.52.0/lib/excon/connection.rb:272
error_call at uri:classloader:/gems/excon-0.52.0/lib/excon/middlewares/idempotent.rb:27
error_call at uri:classloader:/gems/excon-0.52.0/lib/excon/middlewares/base.rb:11
error_call at uri:classloader:/gems/excon-0.52.0/lib/excon/middlewares/base.rb:11
request at uri:classloader:/gems/excon-0.52.0/lib/excon/connection.rb:272
error_call at uri:classloader:/gems/excon-0.52.0/lib/excon/middlewares/idempotent.rb:27
error_call at uri:classloader:/gems/excon-0.52.0/lib/excon/middlewares/base.rb:11
error_call at uri:classloader:/gems/excon-0.52.0/lib/excon/middlewares/base.rb:11
request at uri:classloader:/gems/excon-0.52.0/lib/excon/connection.rb:272
request at uri:classloader:/gems/fog-xml-0.1.2/lib/fog/xml/sax_parser_connection.rb:35
request at uri:classloader:/gems/fog-xml-0.1.2/lib/fog/xml/sax_parser_connection.rb:-1
request at uri:classloader:/gems/fog-xml-0.1.2/lib/fog/xml/connection.rb:7
_request at uri:classloader:/gems/fog-aws-1.4.0/lib/fog/aws/storage.rb:612
_request at uri:classloader:/gems/fog-aws-1.4.0/lib/fog/aws/storage.rb:-1
request at uri:classloader:/gems/fog-aws-1.4.0/lib/fog/aws/storage.rb:607
get_bucket at uri:classloader:/gems/fog-aws-1.4.0/lib/fog/aws/requests/storage/get_bucket.rb:43
get at uri:classloader:/gems/fog-aws-1.4.0/lib/fog/aws/models/storage/directories.rb:21
all at uri:classloader:/gems/fog-aws-1.4.0/lib/fog/aws/models/storage/files.rb:30
each at uri:classloader:/gems/fog-aws-1.4.0/lib/fog/aws/models/storage/files.rb:51
list_files at uri:classloader:/gems/sluice-0.4.0/lib/sluice/storage/s3/s3.rb:69
send_to at uri:classloader:/gems/contracts-0.11.0/lib/contracts/method_reference.rb:43
call_with at uri:classloader:/gems/contracts-0.11.0/lib/contracts/call_with.rb:76
block in redefine_method at uri:classloader:/gems/contracts-0.11.0/lib/contracts/method_handler.rb:138
block in initialize at uri:classloader:/emr-etl-runner/lib/snowplow-emr-etl-runner/emr_job.rb:148
select at org/jruby/RubyArray.java:2564
initialize at uri:classloader:/emr-etl-runner/lib/snowplow-emr-etl-runner/emr_job.rb:146
send_to at uri:classloader:/gems/contracts-0.11.0/lib/contracts/method_reference.rb:43
call_with at uri:classloader:/gems/contracts-0.11.0/lib/contracts/call_with.rb:76
block in redefine_method at uri:classloader:/gems/contracts-0.11.0/lib/contracts/method_handler.rb:138
run at uri:classloader:/emr-etl-runner/lib/snowplow-emr-etl-runner/runner.rb:106
send_to at uri:classloader:/gems/contracts-0.11.0/lib/contracts/method_reference.rb:43
call_with at uri:classloader:/gems/contracts-0.11.0/lib/contracts/call_with.rb:76
block in redefine_method at uri:classloader:/gems/contracts-0.11.0/lib/contracts/method_handler.rb:138
at uri:classloader:/emr-etl-runner/bin/snowplow-emr-etl-runner:41
load at org/jruby/RubyKernel.java:979
at uri:classloader:/META-INF/main.rb:1
require at org/jruby/RubyKernel.java:961
(root) at uri:classloader:/META-INF/main.rb:1
at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
ERROR: org.jruby.embed.EvalFailedException: (Socket) Socket closed (OpenSSL::SSL::SSLError)

We are running SnowplowEtlEmrRunner on a t2.medium EC2 instance type. EMR job is not getting started for past few days consistently. Strangely, retrying multiple times makes it work.

Please help us with this/.

6

@rahul, see EMR Job - Failing with SSL handshake? if it helps.