Does snowplow elasticsearch sink 0.7.0 support elasticsearch 2.3 and 5?

For engineers Storage targets
1

because these documents

recommend 1.4 instead of 2.3

Thank You,

2

Hi @ChocoPowwwa, officially no, the sink does not support Elasticsearch 2.3. By us it has only been tested on Elasticsearch 1.4-1.5.

With the HTTP client it should be possible to send events to all 1.x Elasticsearch clusters. I have tested it briefly with a 2.3.4 cluster but not extensively and not in a production environment - it was able to successfully sink events to the cluster however.

So please do test the sink with a 2.3.4 cluster and let us know if any issues crop up!

The other solution to get the sink working with a 2.x cluster would be to build the sink manually with this dependency bumped to 2.0.3 rather than 1.0.3. See the Jest Github for more version details.

Hope this helps,
Josh

3

yes i’ve tried that, that’s why i’m asking in this forum, and it works fine (tested locally) i haven’t seen any problem yet, so far

4

Hi @ChocoPowwwa, that is great to hear! However as the library we are using to support HTTP sending does not officially support the 2.x series we cannot at this time state that it supports that series.

That being said if you are not having any issues I cannot see any reason not to use it!

Thanks,
Josh

5

I think some code changes might be required for 2.3.
ImmutableSettings class is obsolete so code changes are required https://github.com/snowplow/snowplow/blob/master/4-storage/kinesis-elasticsearch-sink/src/main/scala/com.snowplowanalytics.snowplow.storage.kinesis/elasticsearch/clients/ElasticsearchSenderTransport.scala and also the way ClusterHealth is captured is different from 1.* versions .
I already made code changes and got it working. Just to keep in mind kinesis elasticsearch connector has dependency on elasticsearch 1.* version.

I am using kafka in place of kinesis in that way I am liberated from this dependency.

1 Like
6

Hi Devsaik,

Do you have a fork I can look at to see your changes? I’m interested in what you changed to make it work to 2.3 and also what you did to use Kafka in place of Kinesis? I’m thinking of also using Kafka as a replacement for the ElasticSearch Sink (maybe it’s a bad idea?)

Thanks for your help!
Stephane

7

Hi @simplesteph @devsaik we are actively working on an Elasticsearch Sink which has full support for ES 1.x and 2.x for both the HTTP and Transport clients. The branch to follow for this progress:

https://github.com/snowplow/snowplow/commits/feature/es-2.x-support

There is also an RC available on Bintray from this link which has been tested on the Amazon Elasticsearch Service (2.3):

https://bintray.com/snowplow/snowplow-generic/snowplow-kinesis-elasticsearch-sink#files

8

Hi @josh, Sorry if I missed any updates. Am not able to get elasticsearch 2.x to work with elasticsearchsink 6.0. Are there any latest versions I should update my system with? Any code changes to be done? Would really appreciate your help.

Thanks,
Manju

9

Hi @manju,

We added support for Elasticsearch 2.x in Release 84. You will need to download the latest Elasticsearch Sink binaries first before being able to sink data to a 2.x or 5.x Elasticsearch Cluster.

You can grab them from here:

Thanks,

Josh

10

Hi @josh

Thanks for your response. I tried this elasticsearch sink binaries, but data did not pass on to storage. Does that work only with the certain version of stream collector / stream enrichment binaries as well? Please let me know. I have the streamcollector-streamenrich versions just like snowplow-mini.

Thanks,
Manju

11

Hey @manju,

You will have to be more specific about what is going wrong than data did not pass on. Can you please provide:

  1. All versions of Kinesis Applications you are currently using
  2. What version of Elasticsearch you are using
  3. Configuration file for your Elasticsearch Sink
  4. Any debug / error logs from running the Sink

Thanks,
Josh