Catching bad events using the Scala SDK


#1

Hi,

I’m trying to make use of the Scala SDK using
http://snowplowanalytics.com/blog/2016/03/23/snowplow-scala-analytics-sdk-0.1.0-released/#json-event-transformer as a guide but I’m not sure how to process bad events. The example code makes use of Scalaz Validation which is a little(!) over my head and is only filtering for successful events afaik.

Can anyone shine a little light or give an example?

Thanks,
Shin


#2

Hi @Shin - is this what you’re looking for? Debugging bad rows in Spark and Zeppelin [tutorial]


#3

As a side-note - the Scala Analytics SDK does not include support for parsing Snowplow bad rows currently. It’s a cool idea though - I’ve added a ticket, #13 Add support for bad rows.


#4

Hey @christophe, it’s not quite I was after but I realise now my question was a little broad; I think you’re pointing to what to do about bad events in s3/kinesis from enrichment.

I’m actually trying to take the good enriched events and ship them off to somewhere else (BigQuery in this case). I’m from a Groovy background so Scala’s really new to me but the Scala SDK’s EventTransformer.transform(line) returns a ValidatedEvent which lead me to think that if the transform failed then those errors would be kept somewhere and I could at least log them.

So the docs say to extract the events you do something like:
val jsons = input.map(line => EventTransformer.transform(line)).filter(_.isSuccess).flatMap(_.toOption)

So I thought I’d be able to do something like:
val jsons = input.map(line => EventTransformer.transform(line)).filter(_.isFailure).flatMap(_.toOption)

To get the failed ones. But I’m not sure if I’m on the right track…


#5

Hi @Shin - ah, I get you, you’re not talking about Snowplow bad rows, you’re asking about handling the failure side of the Scala Analytics SDK’s event transformer.

You are right - this ValidatedEvent is built on top of Scalaz, specifically its Validation applicative functor. This is old but probably still the best tutorial on Validation:

A Tale of 3 Nightclubs