Unstructured event couldn't be extracted


#1

Hi guys,

I’m in the first experiments using Snowplow Mini for tracking pixel and link click events, so noob alert :wink:

Example:

<img src="http://xxx/i?e=ue&cid=53450&nid=2446468&uid=12345678" alt="Snowplow Mini" width="1" height="1" />`
and
`<a href="http://xxx/r/tp2?e=ue&cid=53450&nid=2446468&uid=12345678&u=http%3A%2F%2Fwww.xxx.com%2F%3Fname%3DThomas">Snowplow Mini Link</a>

Kibana returned it as bad result:

{"level":"error","message":"Field [eid]: [2446468] is not a valid UUID"},{"level":"error","message":"Unstructured event couldn't be extracted"}

Then I’ve submited some changes to the schemas and also replaced “eid” to “nid” since “eid” seems a reserved variable for Snowplow.

Current schema for tracking a pixel impression:

{
  "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
  "description": "Schema for open tracking event",
  "self": {
    "vendor": "com.xxx",
    "name": "open",
    "version": "1-0-1",
    "format": "jsonschema"
  },
  "type": "object",
  "properties": {
    "cid": {
      "type": "string"
    },
    "nid": {
      "type": "string"
    },
    "uid": {
      "type": "string"
    }
  },
  "additionalProperties": true
}

Now, after a few tries finding the right schema within Iglu it does still returns:

{“level”:“error”,“message”:“Unstructured event couldn’t be extracted”}

Any help?

Also, since there isn’t any DELETE method in the Iglu schema repository, is there a way to delete previous schemas uploaded?

Thanks!


#2

Hi @T_P to send your own custom Unstructured Events you will need to structure your data a touch differently. The actual event JSON for your unstructured event has to be nested as a json in a key-value pair. The reason you are seeing that it cannot extract the Unstructured Event is that you are missing the ue_pr key which is where your custom event must be stored.


To send as a POST request the data payload should be sent to http://xxx/com.snowplowanalytics.snowplow/tp2 with the following JSON data:

{
  "schema": "iglu:com.snowplowanalytics.snowplow/payload_data/jsonschema/1-0-4",
  "data": [
    "e": "ue",
    "ue_pr": "{\"schema\":\"iglu:com.xxx/open/jsonschema/1-0-1\",\"data\":{\"cid\":\"53450\"}}",
    "tv": "custom",
    "p": "web"
  ]
}

You will also need to provide a Content-Type header of application/json; charset=utf-8


To send as a GET request the querystring should be sent to http://xxx/i? with the following querystring:

e=ue&ue_pr=%7B%5C%22schema%5C%22%3A%5C%22iglu%3Acom.xxx%2Fopen%2Fjsonschema%2F1-0-1%5C%22%2C%5C%22data%5C%22%3A%7B%5C%22cid%5C%22%3A%5C%2253450%5C%22%7D%7D&tv=custom&p=web

The JSON for the ue_pr key has been URL Encoded.


Hope that helps!

Josh


#3

Thank you @josh
Helped for sure!

I’m passing everything as a GET request like suggested:

<img src="http://xxx/i?e=ue&ue_pr=%7B%22schema%22%3A%22iglu%3Acom.xxx%2Fopen%2Fjsonschema%2F1-0-1%22%2C%22data%22%3A%7B%22cid%22%3A%2253450%22%2C%22eid%22%3A%222446468%22%2C%22uid%22%3A%2212345678%22%7D%7D&tv=custom&p=web" alt="Snowplow Mini" width="1" height="1" />

But keeps returning errors:

"errors": [
  {
    "level": "error",
    "message": "error: Verifying schema as iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-* failed: found iglu:com.xxx/open/jsonschema/1-0-1"
  },
  {
    "level": "error",
    "message": "Unstructured event couldn't be extracted"
  }
]

Damn schemas :smiley:

Do I need to wrap the current schema in another one?


#4

Sorry yes you do need another layer of wrapping! This is the primary reason behind all of our Tracking SDKs… simplifies this all a great deal.

It will need to be:

"{\"schema\":\"iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0\",\"data\":[{\"schema\":\"iglu:com.xxx/open/jsonschema/1-0-1\",\"data\":{\"cid\":\"53450\"}}]}"

Sorry about the confusion!


#5

Even before you replied, I’ve tried it, but doesn’t seem to work…

"errors": [
  {
    "level": "error",
    "message": "error: instance type (array) does not match any allowed primitive type (allowed: [\"object\"])\n    level: \"error\"\n    schema: {\"loadingURI\":\"#\",\"pointer\":\"\"}\n    instance: {\"pointer\":\"\"}\n    domain: \"validation\"\n    keyword: \"type\"\n    found: \"array\"\n    expected: [\"object\"]\n"
  }
]

#6

Sorry this one should do it, the unstruct schema does not need an array:

"{\"schema\":\"iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0\",\"data\":{\"schema\":\"iglu:com.xxx/open/jsonschema/1-0-1\",\"data\":{\"cid\":\"53450\"}}}"

If that still does not work let me know!


#7

Hello Josh,

I’m a co-worker of T_P. I’ve tried your proposal, for a click - example:

http://xxx/r/tp2?e=ue&ue_pr={"schema"%3A"iglu%3Acom.snowplowanalytics.snowplow\%2Funstruct_event\%2Fjsonschema\%2F1-0-0"%2C"data"%3A{"schema"%3A"iglu%3Acom.xxx\%2Fclick\%2Fjsonschema\%2F1-0-1"%2C"data"%3A{"cid"%3A"53450"%2C"eid"%3A"2446468"%2C"uid"%3A"12345678"}}}&tv=custom&p=web&u=redirect_url

But keep getting the following error:

"errors": [ { "level": "error", "message": "error: Could not find schema with key iglu:com.xxx/click/jsonschema/1-0-1 in any repository, tried:\n level: \"error\"\n repositories: [\"Iglu Central [HTTP]\",\"Iglu Server [HTTP]\",\"Iglu Client Embedded [embedded]\"]\n" } ]

The json schema is:

{"schema":"iglu:com.snowplowanalytics.snowplow\/unstruct_event\/jsonschema\/1-0-0","data":{"schema":"iglu:com.xxx\/click\/jsonschema\/1-0-1","data":{"cid":"53450","eid":"2446468","uid":"12345678"}}}

Any idea what it might be wrong in the schema?

Thanks!


#8

Hi @vjoxyodo, that could be an issue either with:

  1. If you send the events without it being in the Iglu Server then it will be cached as “bad” and you will need to restart your instance before it will find it again.

  2. The script for uploading in 0.2.0 was incorrectly making the Schemas Private rather than Public. If you grab the updated script from here and upload again it should work.

Hopefully that will get it working for you!

Josh


#9

Hey @josh

We’ve tried your suggestions:

  1. Used the newest version of iglu_server_upload.sh where “?isPublic=true” is setted and updated all existing schemas. Status 200 was returned.

  2. In the Iglu schema repository searched for the vendor prefix and returned all updated schemas above.

Fired an event, data went flagged as “bad” and the output remained exactly the same as before:

{"level":"error","message":"error: Could not find schema with key iglu:com.xxx/click/jsonschema/1-0-1 in any repository, tried:\n level: \"error\"\n repositories: [\"Iglu Central [HTTP]\",\"Iglu Server [HTTP]\",\"Iglu Client Embedded [embedded]\"]\n"}

Is there anything else we can check?

TIA


#10

Hi @T_P did you restart the Snowplow Mini instance to reset the cache?

The Stream Emitter holds an internal cache where if an event is sent with a schema that it cannot find that schema is locked into a bad cache - so we do not waste processing time attempting to find a bad schema over and over. For the moment the only way to reset this cache is to restart the instance.

If you have done the above are you certain that the schema you are sending with the event matches what is in the Iglu Server?


#11

We have “good” data. It’s working!

The restart solved it :smiley:

Thank you for your patience in helping us, @josh


#12

No worries @T_P! Glad it started working in the end.