EDIT: 2021-03-04 16:05 UTC – Initially I suggested that Snowflake Loader inspects the
atomic.events table to determine what columns need to be added, which is incorrect and changes the conclusion as well.
Hi @sphinks , the error message suggests that Snowflake Loader tried to create the column
UNSTRUCT_EVENT_COM_SHIPMENT_FORWARDED_1 but the
ALTER TABLE SQL statement failed because the column already existed in the
The way Snowflake Loader works (simplified to fit the current discussion) is:
- check state of processing manifest
- determine what data needs to be loaded and if any new columns need to be created (because the manifest does not indicate they have been created)
- create those columns
- load the data.
In this case, the column could not be created.
It is safe to rerun in the sense that no data was loaded, so there will be no duplication.
How did it fail in the first place? Hard to say without more information, but you mentioned that this is historical data. Perhaps your manifest does not contain the proper information required by the loader to determine that the column already exists.
If that is the case, when re-running the job it should go through the same process as described above; and it will probably fail again because the manifest would still be telling it that the column needs to be created.
Could you please check and let us know some more details about what you are trying to do: timestamps of runs you want to load (min and max), timestamps of data in your processing manifest table and anything else that seems relevant. We can then try to advise on a way forward.