Shredder not producing shredded-tsv

For engineers
1

Hi,

First of all thanks for the great work on Snowplow! Keep up the great work :wink:

I’m currently facing an issue with auto-migrations. I followed the upgrade guide R32 Upgrade Guide - Snowplow Docs

After this I created a new schema on our private Iglu instance and started using it. Enricher is processing the new events correctly so the schema is OK and available.

I ran the shredder and loader and saw the loader failed because it did not find the JSONPath file for the new event. On inspection I could see that the shredder wrote files for the new event in the shredded-types folder instead of the shredded-tsv folder. I guess this is why the loader is asking for the JSONPath file.

Any ideas why shredder is not producing TSV for this new event type?

Kind regards, Jeroen

2

Hi @jeroen.v,

Thanks for the kind words!

I think your problem is that blacklistTabular is either missing or null, which means pre-R32-behavior or JSON everywhere.

If you populate blacklistTabular with schema criterions that you want to shred into JSON - everything not listed there with be shredded into TSV. So for example if you have schemas:

  • igiu:com.acme/event_one/jsonschema/1-0-0
  • igiu:com.acme/event_one/jsonschema/2-0-0
  • igiu:com.acme/event_two/jsonschema/1-0-0

And you want event_two to be shredded into TSV, your config should look like following:

{
  "blacklistTabular": [
    "igiu:com.acme/event_one/jsonschema/1-*-*",
    "igiu:com.acme/event_one/jsonschema/2-*-*"
  ]
}

That distinction between null (legacy) and [] (all in-TSV) is a bit confusing. Recently we released R35 with hopefully more sensible configuration and many other improvements, so I can remcommend you to have a look at it.

3

Hi Anton, thanks!

That indeed fixed it. I passed the target file only to the loader and not the enricher so my mistake.

I will have a look at the R35 in a few weeks when I find some time. Looks promising!

Kind regards, Jeroen