When working with the Scala Stream Enricher, I noticed that after splitting shards the enricher was not pulling any data from the collector good output stream. Here’s what happened:
- upped capacity of collector output good stream from 1 shard to 2 shards
a. shard-00000000 was closed, and shard-00000001 and shard-00000002 were created
- the enricher was still connected to shard-00000000, and did not connect to either of the new shards
- I had to restart the enricher application for it to connect to the new shards
My question is this: is this the expected behavior?
Assuming it is, I know I can use TRIM HORIZON as the initial starting position to make sure I don’t lose data and all records get processed. However, this does mean I need to restart the application and/or drop existing enricher servers once they’ve caught up and finished processing their (now closed) shards. I wanted to check on this in case there’s a different or better solution!