How to see the data

nando_roz · September 17, 2021, 11:18pm

Hi all,

I created my own schema, upload into iglu server (./igluctl static push --public schemas/com.snowplowanalytics.snowplow/login/ {iglu endpoint} {API KEY}) but when executing my easy example as the image attached, I couldn’t see the data in Elastic and S3 the other events works fine. Should I do something else?

I’ve checked into IGLU RDS and See the schema uploaded

I have already created, validated and uploaded the schema, but I didn’t found where to enable the schema and sending the event in the app to S3 and Elastic good Index.

What should I do to see the data in S3 good enrichment and good index in Elastic Search

Regards,

anton · September 19, 2021, 2:23pm

Hi @nando_roz,

Vendors do not match. In tracking you have com.snowplowanalytics and in RDS you have com.snowplowanalytics.snowplow (with additional snowplow). Vendor names have to match.

Also, I recommend you to use your company’s name for the vendor. Technically, there’s no difference, but it can get confusing once you’ll have tens of schemas.

nando_roz · September 19, 2021, 3:34pm

Hi @anton thanks for your reply. I Will check and back you soon.

With this change, automatically the schema Will papear in the shcema field in S3 good enrichment event when I add the schema field in the good index or should I do something else.

About your suggestion, I Will put another vendor as my company

Thanks by advance

anton · September 19, 2021, 3:50pm

With this change, automatically the schema Will papear in the shcema field in S3 good enrichment event when I add the schema field in the good index or should I do something else.

Most likely yes, but there’s a small heads-up.

If you tracked any events with a new schema before you uploaded the schema - your resolver can cache the absence of the schema, so it doesn’t retry it immediately next time. However, this absence cache is usually short-living (depending on timeout config in your resolver.json) and eventually it will manage to resolve it, just not immediately.

In other words, if you tracked any events with the schema already you have two choices:

wait a little bit - the schema will be resolved
reboot your enrich asset - it will be forced to re-fetch all schemas

If you haven’t tracked it before uploading - it will appear immediately.

nando_roz · September 19, 2021, 4:50pm

@anton Just to confirm, I create a new folder in the schemas directory as imagem below

And the json schema I created like that:

With that I just waiting the sync and the shcmea data will be in this column in the S3 enrichment bucket?

To appear in Elastic, just to add manually this field in the good index or automatically will appear?

The test will be only seeing the data or I can check in other place to see if the event send the data?

nando_roz · September 22, 2021, 4:49pm

@anton could you please help me?

Colm · September 22, 2021, 5:38pm

The data will either go to the good stream, or land in the failed events stream, along with an error message which tells you why it didn’t pass validation (or failed for some other reason).

You can read more, to understand failed events here.

You’ll need to access the failed events, and debug according to the error messages. To do so, the most common options are to set up the S3 loader, or set up the Elasitcsearch loader on the failed events stream.

I would highly recommend, though, that you use either Snowplow Mini or Snowplow Micro to debug your tracking. Both come packaged with easy ways to access your failed events, and will give you a fast feedback loop. In your case, Mini is likely to be most helpful since it’ll closely mirror your production setup.

Generally you’ll use one of those tools to debug any new tracking setup, and the main pipeline’s failed events stream to keep track of production issues.

Hope that’s helpful!

nando_roz · September 22, 2021, 5:54pm

Hi @Colm, thanks for your reply. I will check this.

nando_roz · September 22, 2021, 6:01pm

Hi @Colm
I Got this error:
{
“schemaKey”: “iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0”,
“error”: {
“error”: “ValidationError”,
“dataReports”: [
{
“message”: “$.schema: is missing but it is required”,
“path”: “$”,
“keyword”: “required”,
“targets”: [
“schema”
]
},
{
“message”: “$.data: is missing but it is required”,
“path”: “$”,
“keyword”: “required”,
“targets”: [
“data”
]
},
{
“message”: “$.event: is not defined in the schema and the schema does not allow additional properties”,
“path”: “$”,
“keyword”: “additionalProperties”,
“targets”: [
“event”
]
}
]
}
}

This is the code

function login(){
    snowplow('trackSelfDescribingEvent', {
  event: {
    schema: 'iglu:com.mycompany/login/jsonschema/1-0-0',
    data: {
      product_id: 'jd7ujksd8-djdps89d98v-8klnl899n',
      product_name: 'AZ',
      card_position: 1
          }
        }
      })
    };

mike · September 23, 2021, 6:35am

This looks correct - just to confirm are you including the Javascript 3 tracker or using the previous Javascript 2 tracker? (this will be at the top of the Javascript file you are including as part of the initialisation). I think it may be possible to get this output if you use the v3 Javascript tracker syntax with the v2 tracker library.

nando_roz · September 23, 2021, 11:50am

@mike

Should be it?

Colm · September 23, 2021, 12:19pm

Well spotted by mike. Indeed that’s version 2 of the tracker. Easiest path to resolve is to use version 3 instead - here’s the relevant part of the docs.

Ps. We recommend self-hosting for production, as outlined here. Of course it’s up to you and certainly to get started there’s no harm at all in using the cdn, I just mention it for future reference.

nando_roz · September 23, 2021, 10:41pm

@colm

I change just for test but I will implement as self-hosting

For struct event works fine just for the custom as above
Now I got a new error:

{
“schemaKey”: “iglu:com.conexia/login/jsonschema/1-0-0”,
“error”: {
“error”: “ResolutionError”,
“lookupHistory”: [
{
“repository”: “Iglu Central”,
“errors”: [
{
“error”: “NotFound”
}
],
“attempts”: 8,
“lastAttempt”: “2021-09-23T22:34:34.999Z”
},
{
“repository”: “Iglu Central - Mirror 01”,
“errors”: [
{
“error”: “NotFound”
}
],
“attempts”: 6,
“lastAttempt”: “2021-09-23T22:34:35.210Z”
},
{
“repository”: “Iglu Client Embedded”,
“errors”: [
{
“error”: “NotFound”
}
],
“attempts”: 1,
“lastAttempt”: “2021-09-23T21:42:55.361Z”
},
{
“repository”: “Iglu Server”,
“errors”: [
{
“error”: “RepoFailure”,
“message”: “no protocol: {Collector URL}/api/schemas/com.mycompany/login/jsonschema/1-0-0”
}
],
“attempts”: 8,
“lastAttempt”: “2021-09-23T22:34:34.738Z”
}
]
}
}

mike · September 23, 2021, 10:50pm

This means that this Iglu schema couldn’t be found in any of these repositories.

If this is in a custom Iglu Server repository the URL should be different from the collector URL (different services).

nando_roz · September 23, 2021, 11:12pm

This is stranger because I uploaded the schema.

What do you mean an Iglu Server Repo, cause I used the Quick Start

mike · September 23, 2021, 11:52pm

In your Iglu resolver you’ll need to add the Iglu repo URL rather than the collector URL. The Iglu URL is the one that you pushed your schema to sp-iglu-lb-prod-25555...

Colm · September 24, 2021, 10:32am

That would be the section titled ‘Setting up your Iglu Server’.

If you can’t find the iglu_server_dns_name, it will be somewhere in the output of terraform show for this module. As mike said it’s the same URL that you would’ve used to upload the schema.

nando_roz · September 24, 2021, 1:24pm

@Colm
Sorry, I put above Collector but It was configured as Iglu Server
If i put in browser this URL return the JsonSchema for me:

sp-iglu-lb-prod-XXXXXXXX.sa-east-1.elb.amazonaws.com/api/schemas/com.mycompany/login/jsonschema/1-0-0

{
“schemaKey”: “iglu:com.mycompany/login/jsonschema/1-0-0”,
“error”: {
“error”: “ResolutionError”,
“lookupHistory”: [
{
“repository”: “Iglu Central”,
“errors”: [
{
“error”: “NotFound”
}
],
“attempts”: 8,
“lastAttempt”: “2021-09-23T22:34:34.999Z”
},
{
“repository”: “Iglu Central - Mirror 01”,
“errors”: [
{
“error”: “NotFound”
}
],
“attempts”: 6,
“lastAttempt”: “2021-09-23T22:34:35.210Z”
},
{
“repository”: “Iglu Client Embedded”,
“errors”: [
{
“error”: “NotFound”
}
],
“attempts”: 1,
“lastAttempt”: “2021-09-23T21:42:55.361Z”
},
{
“repository”: “Iglu Server”,
“errors”: [
{
“error”: “RepoFailure”,
“message”: “no protocol: sp-iglu-lb-prod-0000000.sa-east-1.elb.amazonaws.com/api/schemas/com.mycompany/login/jsonschema/1-0-0”
}
],
“attempts”: 8,
“lastAttempt”: “2021-09-23T22:34:34.738Z”
}
]
}
}

nando_roz · September 24, 2021, 10:28pm

@mike

I updated the schema. This allow that the iglu is in the Repo?

I tried to find some documents to solve this problem but I couldn’t found. Is it possible step by step to use unstruct event?

Thanks in advance,

mike · September 25, 2021, 2:02am

I can’t see the host that you are pushing to - but if you are getting a 200 back from the Iglu server that’s a good sign that it’s been created successfully.

Topic		Replies	Views
Understanding if iglu schemas have successfully deployed Troubleshooting	5	1247	January 9, 2019
Schema not found in iglu server Troubleshooting	34	801	December 13, 2023
Important: Changes to Iglu Central's api for schema lists Announcements	1	1422	October 6, 2021
Working with a partial fork of Snowplow Tracking SDKs	7	2699	May 24, 2016
Snowplow Mini - Custom Schemas Snowplow Mini	10	2990	April 14, 2016

How to see the data

Related Topics