We are currently attempting to implement snowplow as an alternative to an older, non-scalable analytical process.
I’ve set up the scala collector, enrichment and sink (gathered from https://github.com/jramos/snowplow-kinesis-redshift-sink, which seemed to be the only way to get kinesis stream data into redshift), but run into a couple of problems:
- Shredding appears to be impossible with this setup: derived contexts now end up as json in the events table, not as intended in i.e. com_snowplowanalytics_com_snoplow_ua_parser_context_1. How can we create a pipeline with the realtime scala & kinesis comination, which allows data to end up in redshift & allow shredding?
Any recommendations for the above 2 problems?
Thanks in advance (and for snowplow in general),