Yesterday, I started receiving the following error messages on the enrichment logs. Apparently both the enrichment and the loader module have problems writing or reading into the respective dynamodb tables.
I am using snowplow/stream-enrich-kinesis:1.4.0
Error log enrichment:
First, I thought that it is a problem of access and that I did not set the permissions correctly. I checked that and it seems to be right. In order to test that. I removed the leaseKey entries in the dynamodb tables and the error disappeared on the subsequent test and the data could pass the pipeline again.
However, after some time the error appears again. Deleting the leaseKeys is not a permanent solution.
What are they used for and how does that problem come about? More importantly how can I solve it?
Note, I have not created the dynamodb tables myself. They were created by the modules on the fly (as it is recommended in the documentation)
We’ve seen similar behavior previously when runtime exceptions were happenning inside enrichment code. But now they all should be get caught. Any chance you can configure Sentry to trace it?
Sentry is a service to track all the errors that are happening accross your components : https://sentry.io
But I’m afraid that in your case it won’t give you more info that what you already have in the logs.
This is surprising that both Stream Enrich and S3 loader fail with Kinesis errors. Have you checked that your Redshift quotas (both for writing and reading) are not reached ?
Quick update. This error seems to appear everytime we are chaning our streams, e.g. renaming, changing encryption etc.
It can be solved by clearing the dynamodb tables of the enrichment and loader module.
Yes, we decided to overcome this problem by setting up a job that clears the dynamodb tables every time the kinesis streams are modified (e.g. renamed, change shard counts etc.).
