ElasticSearch HTTP Sink failing with AWS ES Signing


#1

We’re using AWS ES Service for our index. In the last few days, something changed somewhere and we can’t figure out why.

We’re now receiving AWS signature failure when checking for Cluster Health. NTP is correctly setup. Keys are correct.

The interesting thing is this. I tested this request using Postman. When I add “:443” port to the Host, the request fails on the signature. But if remove, it works well.

I noticed, in the sink log files, that it always mentions the: 443 port - which is configured in the config.yml file. I can’t remove the port from the config.

I wonder if something changed on the AWS side that is now stripping the “:443” from the Host header when calculating the sign key on their side.

[ForkJoinPool-2-worker-1] ERROR com.snowplowanalytics.elasticsearch.loader.clients.ElasticsearchSenderHTTP - Couldn't retrieve cluster health
org.elasticsearch.client.ResponseException: GET https://awsesdomainamehere.us-west-2.es.amazonaws.com:443/_cluster/health/_all: HTTP/1.1 403 Forbidden
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'GET\n/_cluster/health/_all\n\nconnection:close\ncontent-length:\nhost:awsesdomainamehere.us-west-2.es.amazonaws.com\nuser-agent:Apache-HttpAsyncClient/4.1.2 (Java/1.8.0_151)\nx-amz-date:20180129T172557Z\n\nconnection;content-length;host;user-agent;x-amz-date\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20180129T172557Z\n20180129/us-west-2/es/aws4_request\n35b839bb1bdb06baa282ab9ba3e293f4270631bd38f6584e1803bef5cdd39da1'\n"}
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:354)
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:343)
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
	at java.lang.Thread.run(Thread.java:748)