Our recommendation is to use time based indices. Internally we use daily indices which which allow us to easily control the amount of data in the cluster as well as providing the ability to change shard counts overtime if your event volumes change.
The way to do this with the Elasticsearch Sink is to:
- Setup an Alias for a daily index
- Use this alias in your sink configuration
- Update your alias each day after creating your new index
- NOTE: You will need to ensure that your alias only points to 1 index for the sink to be able to work!
While the removal of the TTL involves more management it does make the cluster a lot more efficient at scale as it is not constantly searching for data to expire.
Hope this helps!