ElasticSearch HTTP Sink failing with AWS ES Signing

For engineers AWS real-time pipeline
1

We’re using AWS ES Service for our index. In the last few days, something changed somewhere and we can’t figure out why.

We’re now receiving AWS signature failure when checking for Cluster Health. NTP is correctly setup. Keys are correct.

The interesting thing is this. I tested this request using Postman. When I add “:443” port to the Host, the request fails on the signature. But if remove, it works well.

I noticed, in the sink log files, that it always mentions the: 443 port - which is configured in the config.yml file. I can’t remove the port from the config.

I wonder if something changed on the AWS side that is now stripping the “:443” from the Host header when calculating the sign key on their side.

[ForkJoinPool-2-worker-1] ERROR com.snowplowanalytics.elasticsearch.loader.clients.ElasticsearchSenderHTTP - Couldn't retrieve cluster health
org.elasticsearch.client.ResponseException: GET https://awsesdomainamehere.us-west-2.es.amazonaws.com:443/_cluster/health/_all: HTTP/1.1 403 Forbidden
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'GET\n/_cluster/health/_all\n\nconnection:close\ncontent-length:\nhost:awsesdomainamehere.us-west-2.es.amazonaws.com\nuser-agent:Apache-HttpAsyncClient/4.1.2 (Java/1.8.0_151)\nx-amz-date:20180129T172557Z\n\nconnection;content-length;host;user-agent;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20180129T172557Z\n20180129/us-west-2/es/aws4_request\n35b839bb1bdb06baa282ab9ba3e293f4270631bd38f6584e1803bef5cdd39da1'\n"}
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:354)
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:343)
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
	at java.lang.Thread.run(Thread.java:748)
2

I got the exact same problem.
We are implementing a ELK solution using ES from AWS and logstash in a docker.
We tested that implementation in January and it was working fine.
Now when we were close to release and were testing again, it’s not working any more.
logstash-output-amazon_es pluging create the url ending with the port and a backstash, and that always give the 403 error “The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.”
If I access it with Postman using the same url, I get the same error, but removing the ‘:443/’, it works fine, so it’s nothing wrong with the key and secret I’m using.
I don’t know what changed, but we did create a new ES domain in AWS, since the one used in January was deleted.