Postgres Storage - Download Skips. Variable Name Issue?


#1

I’ve spent some time not being able to understand why when running the storageloader that the download step was not doing anything (I’ve checked each other step and can run the ETL end-to-end if I mimic the download). Having got things working I wanted to query if this is a bug or me missing something obvious)

This is running the snowplow_emr_r88_angkor_wat version.

Looking at the source code from here I have a suspicion that a variable name is making the logic skip over the download process.

It appears this naming convention was changed here:
https://github.com/snowplow/snowplow/commit/9c63a7f1fef322de52115cea5de369f12aca55da#diff-0d171ec4103afdfab2c7679236b00987

To test this theory I amended my config file below to make config[:storage][:download_required] exist and things worked as I’d hoped


#3

Hello @gill,

Yes indeed this is a bug. I created an issue to address it. While bug isn’t fixed - you can use amended config.yml as workaround.

Thank you very much for raising.