Elastic load balancer is unhealthy for scala collector elastic beastalk

For engineers Collectors
1

I deployed the scala collector with elastic beanstalk. I uploaded a zip file with the Procfile, collector jar and the config file in it. I got the message saying Successfully bound to /0:0:0:0:0:0:0:0:8080 (attaching the full message below). But Beanstalk reports the unhealthy status of the ELB (attaching a screenshot below). When I try to ping the URL xxx.ca-central-1.elasticbeanstalk.com:8080/health. I got the following error. curl: (7) Failed to connect to xxxx.ca-central-1.elasticbeanstalk.com port 8080: Operation timed out

I know the ELB listens to port 80 and I wonder if there is any additional setup I need to do on the ELB to make it work? I really appreciate any help!!!

May 27 13:01:12 ip-172-31-3-255 web: [main] INFO com.snowplowanalytics.snowplow.collectors.scalastream.sinks.KinesisSink - Creating thread pool of size 10
May 27 13:01:12 ip-172-31-3-255 web: [main] WARN com.snowplowanalytics.snowplow.collectors.scalastream.sinks.KinesisSink - No SQS buffer for surge protection set up (consider setting a SQS Buffer in config.hocon).
May 27 13:01:12 ip-172-31-3-255 web: [main] INFO com.snowplowanalytics.snowplow.collectors.scalastream.sinks.KinesisSink - Creating thread pool of size 10
May 27 13:01:12 ip-172-31-3-255 web: [main] WARN com.snowplowanalytics.snowplow.collectors.scalastream.sinks.KinesisSink - No SQS buffer for surge protection set up (consider setting a SQS Buffer in config.hocon).
May 27 13:01:13 ip-172-31-3-255 web: [DEBUG] [05/27/2021 13:01:13.052] [main] [EventStream(akka://scala-stream-collector)] logger log1-Logging$DefaultLogger started
May 27 13:01:13 ip-172-31-3-255 web: [DEBUG] [05/27/2021 13:01:13.058] [main] [EventStream(akka://scala-stream-collector)] Default Loggers started
May 27 13:01:13 ip-172-31-3-255 web: [DEBUG] [05/27/2021 13:01:13.540] [main] [AkkaSSLConfig(akka://scala-stream-collector)] Initializing AkkaSSLConfig extension...
May 27 13:01:13 ip-172-31-3-255 web: [DEBUG] [05/27/2021 13:01:13.548] [main] [AkkaSSLConfig(akka://scala-stream-collector)] buildHostnameVerifier: created hostname verifier: com.typesafe.sslconfig.ssl.DefaultHostnameVerifier@40113163
May 27 13:01:14 ip-172-31-3-255 web: [DEBUG] [05/27/2021 13:01:14.645] [scala-stream-collector-akka.actor.default-dispatcher-4] [akka://scala-stream-collector/system/IO-TCP/selectors/$a/0] Successfully bound to /0:0:0:0:0:0:0:0:8080
May 27 13:01:14 ip-172-31-3-255 web: [scala-stream-collector-akka.actor.default-dispatcher-5] INFO com.snowplowanalytics.snowplow.collectors.scalastream.KinesisCollector$ - REST interface bound to /0:0:0:0:0:0:0:0:8080

Beastalk reports the ELB is not healthy
image

2

Typically the deployment of the collector isn’t on ELB - that said there’s nothing to stop you doing so.

I’d check what your security groups are - ideally you want your load balancer to be forwarding traffic from port 80 / 443 to whatever port (in this case 8080) that your application is running on.

3

Thank you so much, Mike. I am totally new to Snowplow and I wonder if you could give me some advice on what’s the best practice to deploy the collector?

4

There’s a few different options depending on your preferred method and familiarity but it’s typically one of:

  • deploying on EC2 instances using a prebaked image / AMI setup with an autoscaling group (behind an application load balancer)
  • deploying on something like AWS Fargate if you’re happy to run the collector as a container (images on Dockerhub) and this gives you the flexibility to use ECS / Kubernetes
  • if you want a managed service (in your AWS account but provisioned for you) something like Snowplow Insights