Collector Accepts Connections, Delays for One Minute, Loops

For engineers Collectors
#1

Hi,

I setup my Snowplow pipeline on AWS back in October '20, and it was working fine until ~March 4th / last week.
Now, no events are getting past the collector, despite the collector appearing to receive GET requests from the origin browser/device.

My javascript tracker is self hosted and responds normally to the browser’s request.
When I setup my collector, it handled SSL/TLS properly.

Now, when the browser/device attempts to send events to the collector, 2 things appear to happen:

  1. The browser responds with an SSL error, despite my collector’s certificate not being close to expiry whatsoever.
  2. On the collector log, the messages loop between “New Connection Accepted” and “Delaying cancellation for one minute” and nothing else.

To no avail, I setup Amazon Certificate Manager and a Load Balancer (ACM doesn’t allow direct install of certs, but rather requires use of an ELB or other integrated service to consider a public certificate as “in-use”).

I am having a tough time figuring out what I am doing wrong.

Edit: the browser error = SSL_ERROR_NO_CYPHER_OVERLAP

#2

Hi @joseph, did you upgrade your collector or change its configuration around March 4th; or did it just stop working out of nowhere?

If you upgraded, did you set up the akka.ssl-config section: stream-collector/config.hocon.sample at master · snowplow/stream-collector · GitHub ?

#3

Hi @dilyan, it happened out of nowhere / I did not upgrade or make changes to my collector at or around March 4th.

Now, I have the collector setup in ECS Fargate behind an application load balancer. I have the load balancer listening for HTTPS requests on port 8080 (SSL cert managed by ACM) and forwarding them to a target group (ECS Tasks on the same VPC).

I am still receiving the same error as described above (New connection accepted…Delaying cancellation for 1 minutes…etc). The browser is now showing a 502 error though, instead of the SSL error from before.

Edit: If I have the collector in Fargate behind a Load Balancer (i.e. TLS/SSL is taken care of upstream of the collector), how might I account for this in my config.hocon file?

#4

Hi @joseph,

The setting to disable the collector ssl termination is:

 ssl {
    enable = false
  }

The full example of the config is at stream-collector/config.hocon.sample at master · snowplow/stream-collector · GitHub

The 502 might also be due to how the load balancer is configured to perform the health check, the health endpoint might be wrong.

Regards