Our instances of the Elasticsearch Loader are showing some errors that say failed with message Limit of total fields [1000] has been exceeded. This is preventing new events to be loaded. We know this is not a problem of the Loader but a Elasticsearch configuration limit. We also know that we can increase the limit, but after reading about the topic it seems that it’s better to properly understand what’s happening and treat the disease and not just the symptom.
After reviewing the number of fields in the Elasticsearch index, we don’t see how we are hitting that 1000 fields limit. At this point we don’t have so many self describing events nor so many contexts.
Has someone here come across a similar problem? Is there anything we’re not taking into account?
How many fields do you have now out of interest? And I’m guessing it’s on the good index rather than the bad index?
If you have a few self-describing events this can potentially add up quite quickly as properties / sub properties are enumerated.
One solution may be to give people the option of selecting flattened mappings for some fields (which would avoid hitting that limit) but it comes at the expense of some querying functionality and behaviour - so sometimes bumping the limit might be more appropriate.
@danielsepulvedab we have run into this several times when you start to have several hundred self-describing events / context defined. You can bump this limit up pretty easily though when you create an index - this will involve you setting up your own index creation / rotation system though.
Having too many fields will put strain on your cluster especially at high volume but if you can account for that it should be fine to increase the field limit.
Hey Mike and Josh, sorry for the late reply and thank you so much for your answers.
How many fields do you have now out of interest? And I’m guessing it’s on the good index rather than the bad index?
Yes, it’s in the good index. And we don’t have so many self-describing events (around 14).
At the moment we just increased the limit and everything seems to be working well. I was previously miscounting the number of fields because I was not accounting for the .keyword fields created on each string field, we have a bunch of those so that was making our field count grow quite substantially.