Howdy everyone. I’m currently in the process of setting up a real-time component for a snowplow pipeline. Presently, the elasticsearch cluster is provided by AWS ElasticSearch Service, rather than an independent ES stack in EC2 (or elsewhere). Stream enriching, and sinking to this cluster, work with my barebones/ad-hoc deployment (single stream enrich instance, single elasticsearch-sink instance) after whitelisting the IP of the sink instance in the AWS ES Service’s access policy.
However, future plans are of course to have apps in appropriate ASGs, with automated deployments. I’ve thought of a couple different approaches for handling automated access to the cluster, such as:
- make a static proxy instance with EIP, whitelist this instance in the AWS ES Service, and have any elasticsearch sink app proxy traffic through this instance
- Utilize a NAT gateway with an EIP, whitelist that EIP, and ensure sink instances utilize that NAT gateway
- Bite the bullet and build out our own ES cluster
None of these are optimal, as each adds maintenance overhead. It’d be much more straightforward if the sink apps were able to sign requests to the AWS ES Service endpoint via an iam role (like how reading/writing the kinesis streams is set up already). With that being said…
Is there any capability for the elasticsearch sink app to utilize request signing when sinking to AWS ElasticSearch Service endpoints? Is there a setting I’ve just missed? Sniffing the traffic just shows raw POSTs to the endpoint with no signature headers.
Is there perhaps some other approach that I haven’t though of?