As a side-note - the Scala Analytics SDK does not include support for parsing Snowplow bad rows currently. It’s a cool idea though - I’ve added a ticket, #13 Add support for bad rows.
Hey @christophe, it’s not quite I was after but I realise now my question was a little broad; I think you’re pointing to what to do about bad events in s3/kinesis from enrichment.
I’m actually trying to take the good enriched events and ship them off to somewhere else (BigQuery in this case). I’m from a Groovy background so Scala’s really new to me but the Scala SDK’s EventTransformer.transform(line) returns a ValidatedEvent which lead me to think that if the transform failed then those errors would be kept somewhere and I could at least log them.
So the docs say to extract the events you do something like: val jsons = input.map(line => EventTransformer.transform(line)).filter(_.isSuccess).flatMap(_.toOption)
So I thought I’d be able to do something like: val jsons = input.map(line => EventTransformer.transform(line)).filter(_.isFailure).flatMap(_.toOption)
To get the failed ones. But I’m not sure if I’m on the right track…
Hi @Shin - ah, I get you, you’re not talking about Snowplow bad rows, you’re asking about handling the failure side of the Scala Analytics SDK’s event transformer.
You are right - this ValidatedEvent is built on top of Scalaz, specifically its Validation applicative functor. This is old but probably still the best tutorial on Validation: