Elasticsearch Loader unexpected exception

For engineers AWS real-time pipeline
1

Having this in ES good event loader logs. Here’s my latest config:

source = kinesis

sink {
  good = elasticsearch
  bad = kinesis
}

enabled = good

aws {
  accessKey = iam
  secretKey = iam
}

kinesis {
  initialPosition = TRIM_HORIZON
  maxRecords = 10000
  region = eu-central-1
  appName = "snowplow-loader-good"
}

streams {
  inStreamName = enriched_good
  outStreamName = enriched_bad

  buffer {
    byteLimit = 5000000
    recordLimit = 1000
    timeLimit = 1000
  }
}

elasticsearch {
  client {
   endpoint = "127.0.0.1"
    port = "9200"
    maxTimeout = 10000
    ssl = false
  }

 aws {
    signing = false
    region = eu-central-1
  }

  cluster {
    name = db
    index = "<good-{now/d}>"
    clusterType = enriched_good
  }
}

Been experimenting with various buffer sizes, but still ended up with this exception. Running ES 6.2.2 cluster.
Weirdly, this started happening after my failed attempt to utilize amazon-kinesis-scaling-utils - it incorrectly scaled down my stream aggresively. Does anyone have any guidelines using this tool with snowplow realtime pipeline?

Also, what is a sensible buffer size for ES loader if I run let’s say 6 shard enrich stream. Do people run loaders and enrichers on separate ec2 instances and scale them according to shard count?

2

As in most times here, replying to myself in case anyone else runs into such issue.
We were running a fork from 0.10.0-rc2 and forking 0.10.1 stable instead seems to have fixed the issue.