Some context about my setup: I am running EmrEtlRunner r119 in stream enrich mode on a persistent cluster loading into Redshift. It is running every 20 minutes with a lock to ensure only one run at a time. Runs are failing at the
rdb_load step at least once or twice a week with this error:
ERROR: Data loading error [Amazon](500310) Invalid operation: could not complete because of conflict with concurrent transaction; Following steps completed: [Discover,Load]
Discover and Load always complete and the transaction error occurs at the Analyze stage.
I have been unable to find the cause of the transaction error. The Redshift console reports all queries completed successfully and
STL_TR_CONFLICT is empty. The best I could find is this AWS thread that says that queries that complete in less than a second may not be logged.
Is this an error anyone has run into before? More specifically: what exactly is happening in the analyze stage? What is being run that could have a transaction failure and is there a way for me to get more detailed logs on the progress of the