StorageLoader started consistently failing in our production Snowplow pipeline since we have moved the processing to a different set of instances. Here’s how it looks like:
The following have changed:
- Pipeline now uses Airflow instead of Luigi for orchestration (should not matter)
- StorageLoader runs in a separate VPC and talks to Redshift through VPC peering
- EC2 instances running the Docker image use a different AMI
After a lot of trial and error and searching the interwebs, I stumbled upon the following issue which is my best bet right now: http://docs.aws.amazon.com/redshift/latest/mgmt/connecting-drop-issues.html
Loading works flawlessly when executed from my MacBook so it’s definitely an environmental / configuration issue. I’ll keep trying but wanted to create a topic so that I can a) hear from anyone else who faced the same issue b) document the solution once I find it.