Duplicates due to failure on Storage Loader?


#1

Hey all!

Our Storage Loader job failed today due to a inconsistent table DDL. We fixed the issue, but it got me wondering if running Storage Loader again would create duplicates as dozens of tables were COPY’ed into Redshift before the failure occured.

Perhaps we should COPY, then immediately move the corresponding files to the archive bucket to avoid this?

Cheers


#2

Hi @bernardosrulzon - all of the COPY statements are in a single transaction, so any DDL or similar issues will cause all of the table loads to rollback. Is that what you were asking?


#3

Ah, awesome! That was exactly the point. Thanks!