Hi @rahul - thanks for the additional information.
In your situation, I would migrate to sending all events to the real-time pipeline. You would need to make sure you’d provisioned the real-time pipeline properly to cope with your full event volume, but then you’d simply need to update the CNAME in front of your existing batch pipeline to redirect all existing data through your real-time pipeline. You could setup the real-time pipeline to load the event data into the same data store that the current batch pipeline feeds (e.g. Redshift, Snowflake) so that there’d be no gaps in your data as you do the migration.
You’d then have the ability to extend real-time use cases to your full data set.
I am conscious though that you wrote:
It would be little difficult to move from batch to realtime immediately. We can think about this in near future.
Can you elaborate a bit more there - what difficulties do you envisage?