After a bit of a hiatus, we are pleased to announce a new release of our Snowplow Inspector extension for Chrome - and slightly less pleased to announce some additional fixes for regressions in that new version. Whoops!
There’s not a lot immediately visible here, but a surprising amount of work has gone into this version, and we have a few new features to go through, but also a bunch of fixes for behaviour that’s slowly drifted towards “broken” as various browser APIs changed over time.
New Feature: Schema Management
This is the big highlight of v0.2.16/17! For a long time now, Snowplow Inspector has supported integration with Schema Registries to allow validation of your Self-Describing JSON fields in Entities (aka Contexts) and Self-Describing Events (aka Unstructured Events).
Over time, the ways to host Iglu schemas have grown beyond static web server or S3 buckets, and become more flexible with Iglu Servers. This involved needing an API key to communicate with the server and access Iglu schema definitions. This was supported, but the configuration syntax was unintuitive and arcane to set up, which made it hard to properly use this feature. It was also squirrelled away in the Extension Options page which some users found pretty hard to find.
More recently, Snowplow Insights customers not using the Open Source version of Snowplow will have gotten access to the new Data Structures API for managing schemas; but unfortunately Snowplow Inspector was incompatible with this API, meaning you’d need to do some detective work (and probably wake up Snowplow’s ops team with some funky GraphQL queries) to find and access the underlying Iglu Server if you wanted your schema validation to be up-to-date.
All this changes with the new release! The main toolbar now allows you to “Manage Schemas”, which allows you to let the extension know of multiple Registries, and as it encounters new Self-Describing JSON values, it will communicate with them to find the corresponding schema and validate the payload for you.
You’re now able to add registries via the main interface (rather than the Options page), and we support all of:
- Data Structures API
- Iglu Server API
- Static Registries
- Local Registries
If you already had your registries set up in older versions, they should be migrated into registries in the new version.
The Schema Management page will now also fetch schemas from all your registries, allowing you an easy way to search and see all the schemas available to you in one place, including the public ones from the default Iglu Central registry.
New Feature: Static Registry manifests
Static Iglu Registries are an easy, low maintenance way to make your schemas accessible to your Snowplow pipeline. Unfortunately, they are tough to work with from a tool perspective because they lack discoverability; there’s no way to know, given a registry URI, which schemas it has available! If you want to do something simple like get all the schemas known by a registry (like we now do in Schema Manager), this isn’t really possible, your only option is to request the schemas Just In Time, as you process events, and hope one of your registries has the schema in question – and if it doesn’t, you still have to keep trying in the future, just in case it doesn’t have it yet.
Schema manifests try to fix this problem, by listing the schemas available in a static registry in a simple-to-parse JSON format. When you define a Static Registry you can optionally provide a URL for a Manifest file. The manifest file lists all the schema URIs available in that registry.
The format isn’t fixed, and at the moment we support:
- Iglu Server API-style ([“iglu:…”])
- Github API-style ({“tree”: [{“path”: “vendor/name/jsonschema/version”}]})
- Gitlab API-style ([{“path”: “vendor/name/jsonschema/version”}])
- Variants of these, but with “files” or “paths” instead of “tree”
By default, Iglu Central comes configured with a manifest listing the schemas available via the Github repo. So don’t be surprised when the extension asks for permission to do so!
Contribute
This release changed a rather large amount of the internals of the extension, you may notice bugs or other odd behaviour – please report them or let us know! Got feedback, feature requests, or just want to say hi? We’d love to hear those, too!
The extension is Free Open Source Software, development happens over on Github patches and other contributions are very welcome!
(There are also more fixes + functionality that you can find in the full blog post here!)