Is it possible to load to both elastic search and redshift in aws streaming pipeline?


#1

Worried about consuming from kinesis good when I have the s3 loader pulling from there. Is this possible without custom code?


#2

@dbuscaglia, sure, S3 Loader is just another consumer/subscriber. As long as all the consumers are working OK (and respect each other) that should not cause any problem.

The diagrams in this post show multiple consumers including Redshift load with Lambda architecture.

Here’s some guide that explains handling consumers issue that could clarify this more: https://docs.aws.amazon.com/streams/latest/dev/troubleshooting-consumers.html#consumer-app-reading-slower.