SASL Kafka Configuration - AWS MSK - Resolved

Hi All,

I am using AWS MSK for Managed Kafka and trying to run Snowplow. I get the following error:

[scala-stream-collector-akka.actor.default-dispatcher-5] INFO com.snowplowanalytics.snowplow.collectors.scalastream.KafkaCollector$ - REST interface bound to /0.0.0.0:80
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node -1 ([MYCLUSTERHOST](http://MYCLUSTERHOST.kafka.us-east-1.amazonaws.com)/10.44.1.27:9096) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.

My Config is:

sink {
  enabled = kafka
  brokers = "MYCLUSTERHOST.kafka.us-east-1.amazonaws.com:9096"
  retries = 0

  producerConf {
    "sasl.jaas.config" = "org.apache.kafka.common.security.plain.PlainLoginModule required username='myuser' password='mypassword';"
    "security.protocol" = "SASL_SSL"
    "sasl.mechanism" = "SCRAM-SHA-512"
  }
}

I am able to produce and consumer with these details and network connectivity exists e.g.

nc -zv MYCLUSTERHOST.kafka.us-east-1.amazonaws.com 9096
Connection to MYCLUSTERHOST.kafka.us-east-1.amazonaws.com 9096 port [tcp/*] succeeded!

Any ideas. I saw a previous post similar issue - but that was a typo they had on sasl.mechanism. Mine is set correctly.

Best,
Wayne

Answering my own question here - user error.

The issue was I didn’t update sasl.jaas.config from plain.PlainLoginModule to scram.ScramLoginModule. Only when I compared my Kafka Connect Configs did I notice difference

2 Likes

Nice, good to hear you got it sorted. Sorry we didn’t get to this before, but thanks for following up!