Loading Redshift from S3 in a different region?


#1

Hi all,

Quick tip! If you have deployed your collector into a different region than your Redshift cluster, the Storage Loader step is probably taking much more time to complete.

Our (clojure) collector is deployed in São Paulo, rotating logs into a sa-east-1 S3 bucket. We have recently learned that moving the raw files to a us-east-1 S3 bucket (and using us-east buckets throughout the process) resulted in ~50% shorter Storage Loader time.

Cheers,
Bernardo


#2

Thanks for sharing @bernardosrulzon!


#3

@alex Would it make sense to add a feature to SL that moves files to a bucket in the same region as Redshift automatically?


#4

Hi @bernardosrulzon - good idea, I have updated this thread name and added this ticket:

https://github.com/snowplow/snowplow/issues/3168