I’d prefer to use IAM based credentials if at all possible for all stages of the Snowplow pipeline, to avoid having configuration file with hardcoded, plaintext key/secret settings.
I’ve been trying to run the storage loader with the key and secret specified as “iam”, after looking through the source code to verify that’s the most likely way to signal that to the code. But after extensive debugging, including an AWS ticket to review the S3 logs from the errors, I’m starting to think that’s not supported.
While I see S3 API’s being sent, including the correct IAM/instance access key and some sort of secret, the API calls fail with the following error and the program exits.
— cut here —<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n
InvalidAccessKeyIdThe AWS Access Key Id you provided does not exist in our records.ASIAJWUD6PFHBCXXXXXXetc...
— cut here —
Is there a way to get the storage loader to leverage IAM roles assigned to the instance the code is running on? Or do I have to revert to putting key/secret into for an IAM user into the config file?