After some research on this forum and on github I know that there is some plans to create a loader able to load stream enriched events into Redshift, but curently I would like to achieve kind of the same thing with the plain of jars release (r89).
This is our current pipeline (we currently don’t use the result of stream enriched)
This is what I would like to do:
Basically as we are enriching in near real time I just would like to shred and load the stream enriched data into Redshift.
I managed to do it by using the GZIP sink on the Stream enrich good stream, nevertheless it required some tweaks because by default the EmrEtlRunner expect enriched events files to follow a pattern containing
part- which is not the case if you use one of the S3 Kinesis sink.
Is there any plan in supporting this flow, or will we see a stream shredder which could then have its own sink to load the data into Redshift afterwards?
Thanks in advance!