IAM setup for Scala/Streaming/Kinesis

For engineers
1

We’re in the process of migrating from an out of date release of the Clojure Snowplow. And we’d like to jump directly to the latest Scala/Kinesis version of the package.

The roadblock I’ve hit is defining the IAM permissions I need to create than assign to the collector instances. The documentation I’ve found so far only talks about creating users, not policies/roles. And the permissions listed in those docs are very broad, with things like “iam:", "s3:”, and “redshift:" applied to "Resource:”.

What I really need is a more targeted list of permissions that the various parts of the pipeline need so I can build policies, the aggregate those to roles, and apply the appropriate roles to the instances handling the stages of the pipeline.

Any pointer to anything related to IAM permissions needed by the Scala/Kinesis version of the Snowplow software would be appreciated.

-jj

2

For development/testing I’ve assigned full access but I’m going back now to lock things down.

I guess it’s done this way because Snowplow’s best-practise is to use a subaccount and a role to keep things apart.

But I’m trying to deploy Snowplow in the same ECS cluster to make better use of resources I’m struggling with this too.

3

This is what I’ve got so far and it seems to work. Probably don’t need the GETs for S3.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionAcl",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:PutObjectVersionAcl"
            ],
            "Resource": [
                "arn:aws:s3:::snowplow-enriched-events-dev/*"
            ]
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "dynamodb:*"
            ],
            "Resource": [
                "arn:aws:dynamodb:eu-west-1:*:table/Snowplow*"
            ]
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "kinesis:*"
            ],
            "Resource": [
                "arn:aws:kinesis:*:*:stream/*-dev"
            ]
        }
    ]
}
4

I’m plowing through the process of setting things up too. If we continue to pursue using the Kinesis based Snowplow pipeline I’ll post the final IAM settings that worked for me.

-jj