I’ve got a couple of questions about using the Snowflake loader to load the output of the realtime pipeline via S3 - apologies if these have been answered already elsewhere!
If we’re partitioning the folders by YYYY-MM-DD-HH, is there a risk of a partially processed folder being marked as completed if the snowflake loader run occurs in the whilst the hour’s folder is still being filled
I’ve had a couple of instances where the transformer step has failed (due to an AWS problem), and then the job gets stuck due to new columns already existing. Manually dropping the columns fixes the problem. Is this expected behaviour?