We are using two different pipelines for events tracking. One is for real-time data processing (Stream Enrich to Kinesis to Elasticsearch) and the other one is for batch processing of events (EmrEtlRunner to Redshift).
The number of events getting saved in both of them do not match. Somehow the events in Redshift is more than the number of events in Elasticsearch. Also, there are some events that are not present in Redshift but are in Elasticsearch. I have checked all the
bad kinesis streams and those streams are (almost) empty.
Is there any other way of debugging this problem?