AWSSDK Error with snowplow/fs2-enrich:latest

For engineers Enrichment
1

I’m setting up Snowplow on GCP using FS2-enrich, and getting an AWSSDK error on startup, which is confusing!

The error is:

software.amazon.awssdk.core.exception.SdkClientException: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@62492b73: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@7c63dc2c: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or  system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@4650d95e: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@3b533436: Unable to retrieve region information from EC2 Metadata service. Please make sure the application is running on EC2.]
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
at software.amazon.awssdk.regions.providers.AwsRegionProviderChain.getRegion(AwsRegionProviderChain.java:70)
at software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.regionFromDefaultProvider(AwsDefaultClientBuilder.java:202)
at software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.resolveRegion(AwsDefaultClientBuilder.java:184)
at software.amazon.awssdk.awscore.client.builder.AwsDefaultClientBuilder.finalizeChildConfiguration(AwsDefaultClientBuilder.java:135)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.asyncClientConfiguration(SdkDefaultClientBuilder.java:175)
at software.amazon.awssdk.services.s3.DefaultS3AsyncClientBuilder.buildClient(DefaultS3AsyncClientBuilder.java:28)
at software.amazon.awssdk.services.s3.DefaultS3AsyncClientBuilder.buildClient(DefaultS3AsyncClientBuilder.java:22)
at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.build(SdkDefaultClientBuilder.java:124)
at com.snowplowanalytics.snowplow.enrich.fs2.io.Clients$.$anonfun$mkS3$1(Clients.scala:109)
at flatMap @ com.snowplowanalytics.snowplow.enrich.fs2.io.Clients$.mkS3(Clients.scala:109)
at liftF @ com.snowplowanalytics.snowplow.enrich.fs2.Environment$EitherTOps.resource(Environment.scala:194)
at use @ com.snowplowanalytics.snowplow.enrich.fs2.Main$.$anonfun$run$3(Main.scala:38)
at map @ fs2.concurrent.SignallingRef$.in(Signal.scala:150)
at apply @ com.snowplowanalytics.snowplow.enrich.fs2.config.ConfigFile$.parse(ConfigFile.scala:71)
at apply @ com.snowplowanalytics.snowplow.enrich.fs2.config.ConfigFile$.parse(ConfigFile.scala:71)
at use @ com.snowplowanalytics.snowplow.enrich.fs2.config.ConfigFile$.parse(ConfigFile.scala:71)
at use @ com.snowplowanalytics.snowplow.enrich.fs2.config.ConfigFile$.parse(ConfigFile.scala:71)
at use @ com.snowplowanalytics.snowplow.enrich.fs2.config.ConfigFile$.parse(ConfigFile.scala:71)
at apply @ com.snowplowanalytics.snowplow.enrich.fs2.config.ConfigFile$.parse(ConfigFile.scala:71)
at apply @ com.snowplowanalytics.snowplow.enrich.fs2.config.ConfigFile$.parse(ConfigFile.scala:71)
at use @ com.snowplowanalytics.snowplow.enrich.fs2.config.ConfigFile$.parse(ConfigFile.scala:71)

The config file is:

  # "Gcp" is the only valid option now
  auth = {
    type = "Gcp"
  }
  
  # Collector input
  input = {
    type = "PubSub"
    subscription = "projects/${PROJECT_ID}/subscriptions/in"
  
    # Local FS supported for testing purposes
    # type = "FileSystem"
    # dir = "/var/collector"
  }
  
  # Enriched events output
  good = {
    type = "PubSub"
    topic = "projects/${PROJECT_ID}/topics/enriched-good"
  
    # Local FS supported for testing purposes
    # type = "FileSystem"
    # dir = "/var/enriched"
  }
  
  # Bad rows output
  bad = {
    type = "PubSub"
    topic = "projects/${PROJECT_ID}/topics/encriched-bad"
  
  #  # Local FS supported for testing purposes
  #  # type = "FileSystem"
  #  # dir = "/var/bad"
  }
  
  # Optional, for tracking runtime exceptions
  # sentry = {
  #  dsn = "http:#sentry.acme.com"
  # }
  
  # Optional, period after which enrich assets should be checked for updates
  # no assets will be updated if the key is absent
  assetsUpdatePeriod = "7 days"
  
  # Optional, period after Dropwizard will print out its metrics
  # no metrics will be printed if the key is absent
  metricsReportPeriod = "1 second"

Any idea what’s causing this?

2

Hi @iain,

Are all your enrichments configured to get their assets from GCS / HTTP and not from S3 ?

2 Likes
3

oops - we had a maxmind enrichment that was pointing to S3 that was causing the error.

Thanks @BenB! :slight_smile: