Kinesis LZO S3 Sink 0.5.0 Enable logging


#1

Hi All,

How to enable logging for Kinesis LZO S3 Sink 0.5.0 ?

I have set logging level as DEBUG in the config file however only INFO logs are generated. I get following warning as I start application.

log4j:WARN No appenders could be found for logger (com.amazonaws.AmazonWebServiceClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

I was stuck with a problem for 2 days, I could see data being written to Kinesis by stream collector, but it was not written S3 and without logs I was clueless.
Today I deployed earlier version i.e. Kinesis LZO S3 Sink 0.4.1 and I got error almost immediately that user did not have access to dynamo DB. I was using same config file. Is something changed since 0.4.1 related to logging ?

Thanks in advance.


#3

Hello Rajan,

Kinesis S3 uses a log4j properties file to do its logging.

Prior to version 0.5.0, kinesis-s3 used to incorporate an external log4j.properties file on which we had no control (an artifact of some dependency). This problem has been fixed with 0.5.0.

However, since kinesis-s3 is distributed as an executable jar file, this means we cannot pass JVM arguments (like -Dlog4j.properties) when launching it.

As a result, the only solution for now is to add your own log4j.properties file to this folder https://github.com/snowplow/kinesis-s3/tree/master/src/main/resources and rebuild the fat jar from source. The instructions for building kinesis-s3 can be found here.

The longer term solution is to distribute kinesis-s3 as a normal jar so that people will be able to provide their log4j configuration file when launching:

java -Dlog4j.properties=log4j.properties -jar snowplow-kinesis-s3-0.5.0.jar --config config.hocon


#4

Hello Ben.

Oh I see. Thanks for your help.


#5

You can still execute the binary with java -jar, just examine the first few lines of the executable for embedded shell script arguments it passes to java on launch. Adjust accordingly.


#6

Hello @dashirov-ga,
It works like charm. Thanks. It saved me from rebuilding the application.

Rajan


#7

Happy I could help!