How to download assets for Snowflake loader

For data modelers & consumers Snowflake
1

I try to setup Snowflake loader, but it looks like there is a lot of mess in documentation for the loader. Moreover, it is not possible to download assets from S3 now (get error 403).
So I start here: https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/loaders-storage-targets/snowplow-snowflake-loader/setup/
And come to line:

aws s3 cp s3://snowplow-hosted-assets/4-storage/snowflake-loader/snowplow-snowflake-loader-0.7.1.jar .

Result I get

fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Accroding to git https://github.com/snowplow-incubator/snowplow-snowflake-loader I see that there is a 0.8.0 release. I try call:

aws s3 cp s3://snowplow-hosted-assets/4-storage/snowflake-loader/snowplow-snowflake-loader-0.8.0.jar .

The same issue.

I come to page https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/hosted-assets/ where all hosted assets should be, but there is no mention regarding Snowflake at all.

So how can I get jar assets to pass that step in initial setup of Snowflake loader?

P.S. There is also some mess in documentation. I could find 3 versions of setup: in blog, at docs site and at github wiki (I suppose it is better to have the only one). There is an issue https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/loaders-storage-targets/snowplow-snowflake-loader/upgrade-guide/#upgrading-from-0-6-x-to-0-7-x you can see a link Follow the configuration page and update loader configuration which should go to URL https://docs.snowplowanalytics.com/docs/open-source-components-and-applications/snowplow-snowflake-loader/configuration/ but it redirects to Configuration of Lua tracker (https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/lua-tracker/configuration/).

2

Hi @sphinks,

Thanks for the feedback - we’re fixing those doc pages.

Are you sure you have any AWS credentials configured? Those assets are public and surely there. Moreover, you don’t need to download those assets as outlined in docs - you can use Dataflow Runner (in fact Transformer can be launched only this way).

3

Just realised that the bucket is in eu-west-1 also. So if calling from a different region maybe you get that error.

There is a bucket per-region so using format snowplow-hosted-assets-us-east-1 for whatever region you’re using might also work.

4

Definitely I have AWS creds configured (in case we are talking about cli configuration). I also try to get file via http:

http://snowplow-hosted-assets.s3.amazonaws.com/snowflake-loader/snowplow-snowflake-loader-0.7.1.jar

And

http://snowplow-hosted-assets-us-east-1.s3.amazonaws.com/snowflake-loader/snowplow-snowflake-loader-0.7.1.jar

Still no luck.

<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
<Key>
snowflake-loader/snowplow-snowflake-loader-0.7.1.jar
</Key>
<RequestId>ASEPFWFTCYBHFGDM</RequestId>
<HostId>
wVpjbrmpxkrnWxalJiV7oJKMm9byzNs0BcLWBTrPlw+OVbGqDJM+T+HayLXY+EzQ9GzllXjyMLw=
</HostId>
</Error>

I also confused with running all the stuff via Dataflow Runner. Manual have such line regarding setup of :

Two things you need to create manually are Snowflake database and DynamoDB table. After database is created, you can run setup.

To do this you need to use setup CLI action for Snowflake Loader:

Running dataflow is the last point in setup, so I really confused what should I first. Should I download anything, or should I run setup using dataflow somehow at side of AWS?

UPD. I was using incorrect path to jar. Everything could be downloaded by path: http://snowplow-hosted-assets.s3.amazonaws.com/4-storage/snowflake-loader/snowplow-snowflake-loader-0.7.1.jar

5

UPD. I was using incorrect path to jar. Everything could be downloaded by path: http://snowplow-hosted-assets.s3.amazonaws.com/4-storage/snowflake-loader/snowplow-snowflake-loader-0.7.1.jar

I’m glad it’s solved now. Though I’m sure it should be working with plain S3 - just checked one more time with my personal AWS account.

I also confused with running all the stuff via Dataflow Runner.

Yeah, sorry - in order to run setup you need to download the Snowflake Loader, but to set it up later Dataflow Runner would be better suited.