Reading data from raw stream to both batch and real time

For engineers
1

Hi,
I am new to aws and Snowplow, I have set up a stream collector which pushes events to a Kinesis stream.

I have both a stream enricher and a s3 sink up and running and tested separately, however when I try to run them together to get events to both, then events are only read by one or the other, not both.

How do I go about getting the events to go to both enricher and sink from a single Kinesis stream?

Cheers.

2

Hello @jcc,

If messages go to either applications not both, it might be that you specified the same appName for both stream enrich and your s3 sink configuration.

Otherwise, the two applications will be completely independent in what they are reading.

Also be careful that if you’ve mixed app names you might need to manually delete the associated dynamo db tables to clean up any faulty offsets.

For more information, check out http://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-ddb.html.

1 Like
3

Cheers for your help @BenFradet.

I believe that was exactly the issue.

For the sake of completeness here are the steps I took:
The app names were both the same, so I changed them both to unique names, then deleted the single DynamoDB table associated with the old name.

Then messages went to both the sink and the stream enricher.

Thanks again.