I’ve spent some time not being able to understand why when running the storageloader that the download step was not doing anything (I’ve checked each other step and can run the ETL end-to-end if I mimic the download). Having got things working I wanted to query if this is a bug or me missing something obvious)
This is running the snowplow_emr_r88_angkor_wat version.
Looking at the source code from here I have a suspicion that a variable name is making the logic skip over the download process.
It appears this naming convention was changed here:
To test this theory I amended my config file below to make
config[:storage][:download_required] exist and things worked as I’d hoped