Migrating from 0.8.1 to 0.19.1


#1

Hey there,

I’m trying to dockerize the latest version using your supplied containers.

I’ve updated the configuration with what I think it should look like, but I’m getting the following error when trying to run it.

I would love some pointers as to what I need to change.

The error I’m getting:

docker run -v ${PWD}/config:/snowplow/config enrich --config /snowplow/config/config.hocon --resolver file:/snowplow/config/iglu.json --enrichments file:/snowplow/config/enrichments/ --force-cached-files-download
Exception in thread "main" java.lang.NullPointerException
        at scala.collection.mutable.ArrayOps$ofRef$.newBuilder$extension(ArrayOps.scala:190)
        at scala.collection.mutable.ArrayOps$ofRef.newBuilder(ArrayOps.scala:186)
        at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:246)
        at scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
        at scala.collection.mutable.ArrayOps$ofRef.filter(ArrayOps.scala:186)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$$anonfun$8.apply(KinesisEnrich.scala:210)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$$anonfun$8.apply(KinesisEnrich.scala:207)
        at scala.Option.map(Option.scala:146)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$.extractEnrichmentConfigs(KinesisEnrich.scala:207)
        at com.snowplowanalytics.snowplow.enrich.stream.Enrich$class.parseEnrichmentRegistry(Enrich.scala:175)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$.parseEnrichmentRegistry(KinesisEnrich.scala:53)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$$anonfun$2$$anonfun$apply$2$$anonfun$apply$3.apply(KinesisEnrich.scala:67)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$$anonfun$2$$anonfun$apply$2$$anonfun$apply$3.apply(KinesisEnrich.scala:66)
        at scalaz.Validation$class.flatMap(Validation.scala:139)
        at scalaz.Success.flatMap(Validation.scala:345)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$$anonfun$2$$anonfun$apply$2.apply(KinesisEnrich.scala:66)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$$anonfun$2$$anonfun$apply$2.apply(KinesisEnrich.scala:62)
        at scalaz.Validation$class.flatMap(Validation.scala:139)
        at scalaz.Success.flatMap(Validation.scala:345)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$$anonfun$2.apply(KinesisEnrich.scala:62)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$$anonfun$2.apply(KinesisEnrich.scala:60)
        at scalaz.Validation$class.flatMap(Validation.scala:139)
        at scalaz.Success.flatMap(Validation.scala:345)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$.main(KinesisEnrich.scala:60)
        at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich.main(KinesisEnrich.scala)

My config:

enrich {

  streams {

    in {
      # Stream/topic where the raw events to be enriched are located
      raw = "rawdata"
      maxRecords =  10000
    }

    out {
      enriched = "enricheddata"
      bad = "rawdatabad"

      partitionKey = event_fingerprint
    }

    sourceSink {
      enabled =  kinesis

      region = "eu-west-1"

      aws {
        accessKey = iam
        secretKey = iam
      }

      maxRecords = 10000

      initialPosition = TRIM_HORIZON

      backoffPolicy {
        minBackoff = 1000
        maxBackoff = 60000
      }

      retries = 0

    }

    buffer {
      byteLimit = 4500000
      recordLimit = 500
      timeLimit = 60000
    }

    appName = "snowplow_enrich_progress"
  }

}

#3

Hey @joewarrenmeeks IIRC some version needed the “optional” monitoring section, but it could be empty. Give it a go.

HTH