We are using a bunch of enrichments like geo ip lookup, user agent parsing, ip anonymization and also weather enrichment. Some of the enrichments are essential, because there a legal obligation. Or it is just a simple business rule to have certain information attached to an event, otherwise it is not useful. Some other enrichments are purely optional, and we don’t want the event end up in the bad stream, because the enrichment failed. It may be completely different for another user scenario.
Does it make sense to introduce another boolean parameter in each of the enrichment config like
optional or any other descriptive name?
- False: enrichment failed. Not a problem, we pass the event on anyways.
- True: enrichment failed. The event is broken and we send it to the bad queue
This way companies can assemble their unique combination of enrichments. And they are in full control which failure is severe and which is not.