We’re looking at migrating our Redshift events and derived data across to Snowflake. What’s the best way to achieve this? Would we be best to copy the tables from Redshift, or to use the new Snowflake schema and copy the event data from the original enriched data stored in our archive folder on S3?
Hi @iain,
At least with non-derived data I’d go with re-processing enriched events archive.
Enriched archive is our ultimate source of truth and different storage targets apply own appropriate transformations before loading, which means in Snowflake enriched data structured in a very different way. Thus it would take a lot of efforts (or probably even impossible) to transform data unloaded from Redshift into Snowflake.
Thanks Anton.
Am I right in thinking that if you’re only using Snowflake, then shredding is no long a part of the pipeline, just enrichment?
Hey @iain! Yes, you’re right about that. You can think about shredding as of “DB-specific transformation” I mentioned above. So, shredding is Redshift-specific transformation and Snowflake has its own performed by Snowflake transformer. In other words we’re just swapping RDB Shredder with Snowflake Transformer.