Kinesis-s3 to use acl for copying data to raw in


#1

Hello Team,

I am using kinesis-s3 in qa aws account and it is responsible to move the events to s3 in prod account. Because of this, the moved object in prod s3 have ownership of qa . Is there any way that we can use bucket acl with kinesis-s3 configuration?
Or any other solution?


#2

It doesn’t look like this is current possible using the snowplow-s3-loader as far as I can tell. The AWS Java SDK doesn’t look like the current methods for putObject (even the overloaded ones) that easily accepts an ACL but it may be possible to call putObject and then setObjectAcl after the object has been uploaded to S3.


#3

Thanks Mike for your valuable reply.!!
Can you please elaborate more on
that easily accepts an ACL but it may be possible to call putObject and then setObjectAcl after the object has been uploaded to S3.
or suggest some references?


#4

The code in question (that sinks from a Kinesis to S3) is here which uses the AWS Java SDK. I only had a flick through the documentation but it doesn’t look like we can easily add an ACL to the putObject function(s) but it might be possible to set the object ACL in the code after the object has successfully put to S3.


#5

Hi Mike,
Thanks for the solution. This is what I needed. I have changed the code as per my requirement.
Can you please tell me how to build this code? I am doing this first time.I have tried this link https://travis-ci.org/snowplow/snowplow-s3-loader . Please let me know if I am missing anything.


#6

You can build the Scala code into a fat jar using sbt assembly from the root of the project.


#7

Hi Mike! that worked for me

Now I am setting setObjectAcl after putObject like this-
val acl = new AccessControlList();
acl.grantPermission(new CanonicalGrantee(“xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx”), Permission.FullControl);
client.setObjectAcl(bucket, filename, acl)

I am seeing exception

[RecordProcessor-0000] ERROR com.snowplowanalytics.s3.loader.S3Emitter - S3Emitter threw an unexpected exception
com.amazonaws.SdkClientException: Invalid AccessControlList: missing an S3Owner
at com.amazonaws.services.s3.model.transform.AclXmlFactory.convertToXmlByteArray(AclXmlFactory.java:47)

can you please help me in this?


#8

I resolved above issue . nvm.