Hi @BenB
Thanks for the reply,
I re-edit the post with full stack.
The problem was that dynamoDB was throttling since it didn’t scale the capacity quick enough and then the EMR timeout and failed.
To solve it, i set a very high minimum capacity for DynamoDB (which is expensive ) but it seems like a must to handle this amount of data.
As for the upgrade, we attempted it but with no success. couldn’t get the events to load to snowflake.
I still need to give another try - see related post - Upgraded to snowflake loader 0.8.0 but data is not loaded
Shai