Yesterday, I started receiving the following error messages on the enrichment logs. Apparently both the enrichment and the loader module have problems writing or reading into the respective dynamodb tables.
I am using snowplow/stream-enrich-kinesis:1.4.0
Error log enrichment:
I am using snowplow/snowplow-s3-loader:0.7.0:
Error log loader:
First, I thought that it is a problem of access and that I did not set the permissions correctly. I checked that and it seems to be right. In order to test that. I removed the leaseKey entries in the dynamodb tables and the error disappeared on the subsequent test and the data could pass the pipeline again.
However, after some time the error appears again. Deleting the leaseKeys is not a permanent solution.
What are they used for and how does that problem come about? More importantly how can I solve it?
Note, I have not created the dynamodb tables myself. They were created by the modules on the fly (as it is recommended in the documentation)