Hey @christophe, it’s not quite I was after but I realise now my question was a little broad; I think you’re pointing to what to do about bad events in s3/kinesis from enrichment.
I’m actually trying to take the good enriched events and ship them off to somewhere else (BigQuery in this case). I’m from a Groovy background so Scala’s really new to me but the Scala SDK’s EventTransformer.transform(line) returns a ValidatedEvent which lead me to think that if the transform failed then those errors would be kept somewhere and I could at least log them.
So the docs say to extract the events you do something like:
val jsons = input.map(line => EventTransformer.transform(line)).filter(_.isSuccess).flatMap(_.toOption)
So I thought I’d be able to do something like:
val jsons = input.map(line => EventTransformer.transform(line)).filter(_.isFailure).flatMap(_.toOption)
To get the failed ones. But I’m not sure if I’m on the right track…