[scala] [enrich] exception while sync'ing Kinesis shards and leases


#1

Hi,

I am dealing with an exception trying to run enricher and it is failing with this logs:

[main] INFO com.snowplowanalytics.snowplow.enrich.stream.sources.KinesisSource - Using workerId: snowplow-staging-data-snowplow-enrich-5f9cdfddd4-tk8nw:6f05d1af-cdd0-44ee-bde6-de196d825075
[main] INFO com.snowplowanalytics.snowplow.enrich.stream.sources.KinesisSource - Running: data-snowplow.
[main] INFO com.snowplowanalytics.snowplow.enrich.stream.sources.KinesisSource - Processing raw input stream: snowplow-good
[main] INFO com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator - With failover time 10000 ms and epsilon 25 ms, LeaseCoordinator will renew leases every 3308 ms, takeleases every 20050 ms, process maximum of 2147483647 leases and steal 1 lease(s) at a time.
[main] WARN com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Received configuration for both region name as eu-west-1, and Amazon Kinesis endpoint as https://kinesis.eu-west-1.amazonaws.com. Amazon Kinesis endpoint will overwrite region name.
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initialization attempt 1
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initializing LeaseCoordinator
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Syncing Kinesis shard info
[main] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncTask - Caught exception while sync'ing Kinesis shards and leases

The policy included to the enricher has kinesis full access:

{
    "Sid": "VisualEditor1",
    "Effect": "Allow",
    "Action": [
        "kinesis:*"
    ],
    "Resource": "*"
}

I was trying to catch what is happening but i am a bit loss, anyone could lead to what can be misconfigured?


#2

Finally it was related with a fatal mess of AWS policies and roles. Easy to see with debug activated -Dorg.slf4j.simpleLogger.defaultLogLevel=debug