Snowplow Mini - Custom Schemas

For engineers Snowplow Mini
1

Hey guys,

I’m having trouble using custom schemas within Snowplow Mini. All events that use a custom context end up in the bad pipe, with the following error message:

"error: Could not find schema with key iglu:br.com.getninjas/split_test/jsonschema/1-0-0 in any repository, tried: level: "error" repositories: ["Iglu Central [HTTP]","Iglu Server [HTTP]","Iglu Client Embedded [embedded]"] "

The schemas have been succesfully uploaded with the iglu_server_upload.sh script, and I can see them using the Iglu repository API.

Any ideas why this error is happening?

Thanks!
Bernardo

2

Hey Bernardo,

That is very peculiar! Were these events sent through the pipeline prior to the schemas being available in the Iglu Server? The Stream Enrich application uses the Scala Iglu Client that caches bad hits so you might need to restart the Stream Enrich application for it to look for these schemas again. You can also just restart the whole box if easier.

Failing that it might not be a bad idea to SSH onto the box and check the Schema exists inside the Postgres database.

$guest psql --host=localhost --port=5432 --username=snowplow --dbname=iglu
# Password is 'snowplow'
select * from schemas where name like 'split_test';
3

Hey @josh

Yeah, we tried restarting the box with no luck. The schemas exist inside the postgres database - everything looks fine there.

Anything else we can try?

4

Hey @bernardosrulzon could you try re-uploading the schemas but changing this line to isPublic=true.

5

Hey @josh! Thanks so much - it worked!!

6

Glad to hear it and thanks for finding the bug! Will make a ticket for the next release.

7

In Iglu schema repository, i am creating key pair, what should i pass in vendor prefix

8

Hi @JeetuChoudhary, the vendor prefix is whatever vendor prefix you have in the schemas you want to upload.

For example say you wanted to add the client_session context.

To add this you could make a key pair with the vendor prefix: com.snowplowanalytics.snowplow

However you can also use wildcards in your vendor prefix to allow you to upload any schema with a single api-key. This is what we do in the iglu_server_upload.sh script. The line to look at: https://github.com/snowplow/snowplow-mini/blob/master/resources/scripts/iglu_server_upload.sh#L36

For more information please refer to the technical documentation

Hope that helps!

9

If you would like to update an existing custom schema, it needs to be a PUT vs. POST request in this line:

https://github.com/snowplow/snowplow-mini/blob/master/resources/scripts/iglu_server_upload.sh?_sp=c5ac0ded-7393-4be9-95e2-bb46184da7bd.1460644060138#L53

@josh does it make sense to have the upload script use PUT vs. POST since many users will be using snowplow-mini for debugging schemas, etc?

10

Hi @digitaltouch,

That does make a lot of sense and will no doubt save some headaches! Will raise a ticket for the next release.

This could be toggled with another argument to the script potentially?

11

This is actually covered in the Iglu Repo Syncer ticket here: https://github.com/snowplow/iglu/issues/134

Over time the Snowplow Mini schema loader Bash script and the Iglu Repo Syncer are going to merge into a sub-command in the Iglu CLI tool.