Kinesis LZO S3 Sink 0.5.0 Enable logging


Hi All,

How to enable logging for Kinesis LZO S3 Sink 0.5.0 ?

I have set logging level as DEBUG in the config file however only INFO logs are generated. I get following warning as I start application.

log4j:WARN No appenders could be found for logger (com.amazonaws.AmazonWebServiceClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See for more info.

I was stuck with a problem for 2 days, I could see data being written to Kinesis by stream collector, but it was not written S3 and without logs I was clueless.
Today I deployed earlier version i.e. Kinesis LZO S3 Sink 0.4.1 and I got error almost immediately that user did not have access to dynamo DB. I was using same config file. Is something changed since 0.4.1 related to logging ?

Thanks in advance.


Hello Rajan,

Kinesis S3 uses a log4j properties file to do its logging.

Prior to version 0.5.0, kinesis-s3 used to incorporate an external file on which we had no control (an artifact of some dependency). This problem has been fixed with 0.5.0.

However, since kinesis-s3 is distributed as an executable jar file, this means we cannot pass JVM arguments (like when launching it.

As a result, the only solution for now is to add your own file to this folder and rebuild the fat jar from source. The instructions for building kinesis-s3 can be found here.

The longer term solution is to distribute kinesis-s3 as a normal jar so that people will be able to provide their log4j configuration file when launching:

java -jar snowplow-kinesis-s3-0.5.0.jar --config config.hocon


Hello Ben.

Oh I see. Thanks for your help.


You can still execute the binary with java -jar, just examine the first few lines of the executable for embedded shell script arguments it passes to java on launch. Adjust accordingly.


Hello @dashirov-ga,
It works like charm. Thanks. It saved me from rebuilding the application.



Happy I could help!