Hey all!
Our Storage Loader job failed today due to a inconsistent table DDL. We fixed the issue, but it got me wondering if running Storage Loader again would create duplicates as dozens of tables were COPY’ed into Redshift before the failure occured.
Perhaps we should COPY, then immediately move the corresponding files to the archive bucket to avoid this?
Cheers