Hello,
We are deploying enrich module on AWS Fargate with the pushing the enrich configuration below to ECR:
enrich {
streams {
in {
# Stream/topic where the raw events to be enriched are located
raw = ${ENRICH_STREAMS_IN_RAW}
}
out {
# Stream/topic where the events that were successfully enriched will end up
enriched = ${ENRICH_STREAMS_OUT_ENRICHED}
# Stream/topic where the event that failed enrichment will be stored
bad = ${ENRICH_STREAMS_OUT_BAD}
# Stream/topic where the pii tranformation events will end up
pii = ${ENRICH_STREAMS_OUT_PII}
# How the output stream/topic will be partitioned.
# Possible partition keys are: event_id, event_fingerprint, domain_userid, network_userid,
# user_ipaddress, domain_sessionid, user_fingerprint.
# Refer to https://github.com/snowplow/snowplow/wiki/canonical-event-model to know what the
# possible parittion keys correspond to.
# Otherwise, the partition key will be a random UUID.
# Note: Nsq does not make use of partition key.
partitionKey = event_id
}
# Configuration shown is for Kafka, to use another uncomment the appropriate configuration
# and comment out the other
# To use stdin, comment or remove everything in the "enrich.streams.sourceSink" section except
# "enabled" which should be set to "stdin".
sourceSink {
# Sources / sinks currently supported are:
# 'kinesis' for reading Thrift-serialized records and writing enriched and bad events to a
# Kinesis stream
# 'kafka' for reading / writing to a Kafka topic
# 'nsq' for reading / writing to a Nsq topic
# 'stdin' for reading from stdin and writing to stdout and stderr
enabled = kinesis
# Region where the streams are located (AWS region, pertinent to kinesis sink/source type)
# region = {{region}}
region = ${ENRICH_STREAMS_SOURCE_SINK_REGION}
## Optional endpoint url configuration to override aws kinesis endpoints,
## this can be used to specify local endpoints when using localstack
# customEndpoint = {{kinesisEndpoint}}
#customEndpoint = "localstack:4566"
## Optional endpoint url configuration to override aws dyanomdb endpoints for Kinesis checkpoints lease table,
## this can be used to specify local endpoints when using Localstack
# dynamodbCustomEndpoint = "http://localhost:4569"
#dynamodbCustomEndpoint = "localstack:4566"
# Optional override to disable cloudwatch
# disableCloudWatch = true
# disableCloudWatch = ${ENRICH_DISABLE_CLOUDWATCH}
# AWS credentials
# If both are set to 'default', use the default AWS credentials provider chain.
# If both are set to 'iam', use AWS IAM Roles to provision credentials.
# If both are set to 'env', use env variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
aws {
accessKey = default
secretKey = default
}
# GCP credentials
# Either provide path to service account file or set environment variable GOOGLE_APPLICATION_CREDENTIALS
# gcp {
# creds = {{googleApplicationCredentials}}
# creds = ${GOOGLE_APPLICATION_CREDENTIALS}
# }
# Maximum number of records to get from Kinesis per call to GetRecords
maxRecords = 100
# LATEST: most recent data.
# TRIM_HORIZON: oldest available data.
# "AT_TIMESTAMP": Start from the record at or after the specified timestamp
# Note: This only effects the first run of this application on a stream.
# (pertinent to kinesis source type)
initialPosition = TRIM_HORIZON
And we get the following error in CloudWatch after the ECS tasks get terminated automatically after provisioning:
[main] WARN com.networknt.schema.JsonMetaSchema - Unknown keyword exclusiveMinimum - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
[main] WARN com.amazonaws.http.AmazonHttpClient - SSL Certificate checking for endpoints has been explicitly disabled.
[main] WARN com.amazonaws.http.AmazonHttpClient - SSL Certificate checking for endpoints has been explicitly disabled.
[main] WARN com.amazonaws.http.AmazonHttpClient - SSL Certificate checking for endpoints has been explicitly disabled.
Exception in thread "main" java.net.UnknownHostException: d4f5b4e280c4: d4f5b4e280c4: Name or service not known
at java.net.InetAddress.getLocalHost(InetAddress.java:1505)
at com.snowplowanalytics.snowplow.enrich.stream.sources.KinesisSource.run(KinesisSource.scala:147)
at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich$.main(KinesisEnrich.scala:97)
at com.snowplowanalytics.snowplow.enrich.stream.KinesisEnrich.main(KinesisEnrich.scala)
Caused by: java.net.UnknownHostException: d4f5b4e280c4: Name or service not known
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getLocalHost(InetAddress.java:1500)
