We had an issue when trying to parallelize multiple components (enrich, storage s3, storage es). It seems the components put themselves in a paused state, while we added a new one.
The issue didn’t show up with the collector component though.
Here is the stacktrace which showed up when we had two instanciations of a component in parallel :
[RecordProcessor-0005] DEBUG com.amazonaws.services.kinesis.leases.impl.LeaseManager - Lease update failed for lease with key shardId-000000000018 because the lease counter was not 9164846 8:28 [RecordProcessor-0005] INFO com.amazonaws.services.kinesis.leases.impl.LeaseRenewer - Worker 64bc947f0df14e80:-1ff1f730:174bebcb283:-8000 lost lease with key shardId-000000000018 - discovered during update 8:29 [RecordProcessor-0005] ERROR com.amazonaws.services.kinesis.connectors.KinesisConnectorRecordProcessor - com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException: Can't update checkpoint - instance doesn't hold the lease for this shard
Everything is deployed in AWS ECS with Kinesis as queues
Versions of components used:
- snowplow-stream-collector-kinesis : 1.0.0
- snowplow_stream_enrich : 0.13.0
- snowplow_elasticsearch_loader : 1.0.0
- snowplow_s3_loader : 0.7.0
Do you have any feedback on this ? Anyone already had this issue ?
Thanks a lot,