Snowplow 85 Metamorphosis released with beta Apache Kafka support


#1

We have released Kafka support!!

This is just the beginning of the journey and we’re looking forward to working with the community to build this out.


#2

Thank you so much. Am having some trouble when I tried to set this up.
I installed the required version of Kafka, set up the config as mentioned. But the collector is not starting. When I try to open the consumer to see the collector payload, I see only empty terminal.
Below is the last few lines of the collector debug messages of producer starting. Can you please help?

15:27:09.326 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name bufferpool-wait-time
15:27:09.327 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name buffer-exhausted-records
15:27:09.327 [main] DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 1 to Cluster(id = null, nodes = [localhost:9092 (id: -1 rack: null)], partitions = [])
15:27:09.327 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name connections-closed:
15:27:09.327 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name connections-created:
15:27:09.328 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name bytes-sent-received:
15:27:09.328 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name bytes-sent:
15:27:09.328 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name bytes-received:
15:27:09.329 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name select-time:
15:27:09.329 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name io-time:
15:27:09.329 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name batch-size
15:27:09.330 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name compression-rate
15:27:09.330 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name queue-time
15:27:09.330 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name request-time
15:27:09.330 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name produce-throttle-time
15:27:09.331 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name records-per-request
15:27:09.331 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name record-retries
15:27:09.331 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name errors
15:27:09.331 [main] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name record-size-max
15:27:09.332 [main] INFO o.a.kafka.common.utils.AppInfoParser - Kafka version : 0.10.1.0
15:27:09.332 [kafka-producer-network-thread | producer-2] DEBUG o.a.k.c.producer.internals.Sender - Starting Kafka producer I/O thread.
15:27:09.332 [main] INFO o.a.kafka.common.utils.AppInfoParser - Kafka commitId : 3402a74efb23d1d4
15:27:09.332 [main] DEBUG o.a.k.clients.producer.KafkaProducer - Kafka producer started
[DEBUG] [09/26/2017 15:27:09.940] [scala-stream-collector-akka.actor.default-dispatcher-5] [akka://scala-stream-collector/user/IO-HTTP/listener-0] Binding to /0.0.0.0:8083
[DEBUG] [09/26/2017 15:27:09.972] [scala-stream-collector-akka.actor.default-dispatcher-2] [akka://scala-stream-collector/system/IO-TCP/selectors/$a/0] Successfully bound to /0:0:0:0:0:0:0:0:8083
[INFO] [09/26/2017 15:27:09.975] [scala-stream-collector-akka.actor.default-dispatcher-2] [akka://scala-stream-collector/user/IO-HTTP/listener-0] Bound to /0.0.0.0:8083


#3

Hi @manju,

For us to be able to help you here we are going to need a lot more context than just the output from the collector (which appears to be operating correctly and without errors).

Please provide:

  1. The collector configuration file
  2. How you setup Kafka and the associated topics?
  3. How you are actually checking for events downstream of the collector?
  4. What things you have tried to resolve the issue?

Cheers!
Josh