Snowplow Elasticsearch Loader version 0.10.2

#1

Hi,
I have configured Snowplow Elasticsearch Loader version 0.10.2 on one ec2 and that ec2 has aws kinesis full access.
Same Ec2 is having ELK setup.

Using this “https://github.com/snowplow/snowplow-elasticsearch-loader” i have configured
Snowplow Elasticsearch Loader

But when i run the Jar file with my.conf facing some issue.

[root@elk opt]# java -jar snowplow-elasticsearch-loader-http-0.10.1.jar --config my.conf
configuration error: ConfigReaderFailures(KeyNotFound(nsq,Some(ConfigValueLocation(file:/opt/my.conf,1)),Set()),List(KeyNotFound(elasticsearch.aws,Some(ConfigValueLocation(file:/opt/my.conf,32)),Set()), KeyNotFound(elasticsearch.cluster,Some(ConfigValueLocation(file:/opt/my.conf,32)),Set())))

Following is My “my.conf”
[root@elk opt]# cat my.conf

source = kinesis

sink {
good = elasticsearch
bad = elasticsearch
}

enabled = good

aws {
accessKey = iam
secretKey = iam
}

kinesis {
initialPosition = LATEST
maxRecords = 100
region = us-west-2
appName = enrich-stream-good-qa
}

streams {
inStreamName = enrich-stream-good
outStreamName = kinesis-s3-fail-qa
buffer {
byteLimit = 3000000
recordLimit = 200
timeLimit = 180000
}
}

elasticsearch {
client {
endpoint = “127.0.0.1”
port = “9200”
maxTimeout = 10
ssl = false
}
}

#2

Hi @AWSDeven,

You need nsq configuration with dummy values (although it does not make any sense to have them), like:

nsq {
  channelName = "channel"
  nsqdHost = "host"
  nsqdPort = 0
  nsqlookupdHost = "host"
  nsqlookupdPort = 0
} 

I believe this is because of unified config validator.
Cheers,
GE

#3

@grzegorzewald Thanks for the reply… but unfortunately it wont worked. Then i added following config and it is working fine… without your reference it was not possible. Thanks

nsq {
channelName = raw
host = localhost
port = 4150
lookupPort = 4160
}

#4

RESOLVED

Hi All,

Following is the working config. I am able to setup Elasticsearch Loader

1: Download the Elasticsearch Loader jarfile
wget http://dl.bintray.com/snowplow/snowplow-generic/snowplow_elasticsearch_loader_http_0.10.2.zip

2:Unzip downloaded zip file
unzip snowplow_elasticsearch_loader_http_0.10.2.zip

3: Then configure my.conf with your config.
vi my.conf

#------------------------------------------------------------#
source = kinesis

sink {
good = elasticsearch
bad = stderr
}

enabled = good

aws {
accessKey = iam
secretKey = iam
}

kinesis {
initialPosition = LATEST
maxRecords = 100
region = us-west-2
initialTimestamp = “2019-05-08T13:00:00Z”
appName = enrich-stream-good
}

streams {
inStreamName = enrich-stream-good
outStreamName = collector-kinesis-stream
buffer {
byteLimit = 3000000
recordLimit = 200
timeLimit = 180000
}
}

nsq {
channelName = raw
host = localhost
port = 4150
lookupPort = 4160
}

elasticsearch {
client {
endpoint = “127.0.0.1”
port = “9200”
maxTimeout = 10
ssl = false
}

aws {
signing = false
region = us-west-2
}

cluster {
name = elasticsearch
index = snowplow
clusterType = enriched
}
}
#------------------------------------------------------------#


4: Start Elasticsearch Loader jarfile with configuration file

java -jar snowplow-elasticsearch-loader-http-0.10.2.jar --config my.conf

Bingo…
Thank you all of you…