Google Storage Loader - DefaultScalaModule could not be instantiated

For engineers GCP pipeline
1

Hi,

We are attempting to set up Google Cloud Storage Loader using the DataFlow template instructions here

Seeing the following error in DataFlow logs:

java.util.ServiceConfigurationError: com.fasterxml.jackson.databind.Module: Provider com.fasterxml.jackson.module.scala.DefaultScalaModule could not be instantiated

at java.util.ServiceLoader.fail ( java/util/ServiceLoader.java:232 )

at java.util.ServiceLoader.access$100 ( java/util/ServiceLoader.java:185 )

at java.util.ServiceLoader$LazyIterator.nextService ( java/util/ServiceLoader.java:384 )

at java.util.ServiceLoader$LazyIterator.next ( java/util/ServiceLoader.java:404 )

at java.util.ServiceLoader$1.next ( java/util/ServiceLoader.java:480 )

at com.fasterxml.jackson.databind.ObjectMapper.findModules ( com/fasterxml.jackson.databind/ObjectMapper.java:1105 )

at org.apache.beam.sdk.options.PipelineOptionsFactory.<clinit> ( org/apache.beam.sdk.options/PipelineOptionsFactory.java:502 )

at org.apache.beam.sdk.options.ProxyInvocationHandler.<init> ( org/apache.beam.sdk.options/ProxyInvocationHandler.java:132 )

at org.apache.beam.sdk.options.ProxyInvocationHandler.<init> ( org/apache.beam.sdk.options/ProxyInvocationHandler.java:93 )

at org.apache.beam.sdk.options.ProxyInvocationHandler$Deserializer.deserialize ( org/apache.beam.sdk.options/ProxyInvocationHandler.java:817 )

at org.apache.beam.sdk.options.ProxyInvocationHandler$Deserializer.deserialize ( org/apache.beam.sdk.options/ProxyInvocationHandler.java:799 )

at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue ( com/fasterxml.jackson.databind.deser/DefaultDeserializationContext.java:322 )

at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose ( com/fasterxml.jackson.databind/ObjectMapper.java:4674 )

at com.fasterxml.jackson.databind.ObjectMapper.readValue ( com/fasterxml.jackson.databind/ObjectMapper.java:3629 )

at com.fasterxml.jackson.databind.ObjectMapper.readValue ( com/fasterxml.jackson.databind/ObjectMapper.java:3597 )

Below is the terraform config we are using to create the job:

 resource "google_dataflow_job" "gcs_enriched_job" { 
   name = "${var.prefix}-enriched-events-job" 
   template_gcs_path = "gs://sp-hosted-assets/4-storage/snowplow-google-cloud-storage-loader/0.5.0/SnowplowGoogleCloudStorageLoaderTemplate-0.5.0" 
   temp_gcs_location = "gs://${var.prefix}-${var.project_id}-temp-bucket"  
   parameters = { 
     inputSubscription = "projects/${var.project_id}/subscriptions/${var.prefix}-gcs-enriched-server", 
     outputDirectory = "gs://${var.prefix}-${var.project_id}-bucket" 
   } 
   max_workers = 1 
 }

Any tips would be much appreciated. Thanks!