Recent ETL versions & Snowflake collector with old unstruct events


#1

We’re migrating a few of our pipelines across to the newest ETL setup using the Snowflake collector as a target. However one of them has some very old unstructured event types from the pre-Iglu days where it was just a plain JSON.

What will happen to these, and is it possible to migrate them?


#2

Hi @iain - no that won’t work I’m afraid. All those events will fail processing. You’ll need to update those pipelines to use our Iglu-based event technology.


#3

Thanks Alex. Is there anything that can be done with the raw events (from the Clojure collector)? Or given that there are only a handful of different event types, would I be best writing some sort of job to transform the old raw events into new raw events with schemas, then passing them through the ETL as normal?


#4

Yes sure - you can write a job which takes the raw events and fixes the events contained within them to be self-describing.

If you need our Services team to get involved, just ping us an email!