Hi @dadasami,
How long was it taking to load a folder? It seems that visibility timeout is way too low, by the time Loader tries to acknowledge (delete) the message - it is already “lost” to SQS. In 1.0.0 we set it to 5 minutes and acknowledge after the folder has been loaded. As a result if Loader sees the message again - it just considered a duplicate and ignored, but that’s because we have a manifest in 1.0.0, R35 doesn’t have this feature.
I’m still trying to wrap my head around Shredding EMR spark config (IOException: All datanodes ... are bad) - #5 by dadasami to help you to migrate to 1.0.1.
Is there anything wrong with our SQS setup?
I think you can set visibility timeout only in code? If I’m wrong - I definitely recommend you to set to something higher.
We are running the RDBLoader on a single Fargate instance. Does it help if it runs on multiple instances in parallel?
No, certainly not. Loader is designed to be running as a singleton to avoid race conditions and overwhelming Redshift.