The result of my Stream Enrich has more columns than atomic events

1

Hello, guys
I am trying to use Snowplow with Python and React Native trackers, a Scala Stream Collector and a Stream enrich. The result is then loaded via Firehose into S3. The data in S3 is then accessed by Athena to be queried.
The problem is that the data that comes from my Enrichment process does not fit my atomic.events table. It has 2 or 3 columns more than it should have.
I got the atomic.events create table statement from https://github.com/snowplow/snowplow/blob/master/4-storage/redshift-storage/sql/atomic-def.sql

Is there something wrong with the version of the enricher or collector that I’m using? Or am I using an old version for atomic.events?

Thanks in advance

2

@Alexandre_Rayes, indeed, the enriched event has 3 extra properties (when comparing with the shredded event) as per the diagram here (see 2nd image): https://github.com/snowplow/snowplow/wiki/StorageLoader. You can see the structure of a Snowplow enriched event here: https://github.com/snowplow/snowplow/blob/master/3-enrich/scala-common-enrich/src/main/scala/com.snowplowanalytics.snowplow.enrich/common/outputs/EnrichedEvent.scala#L41-L249. Though not sure if that would be the same as data are expected to be loaded to S3 with S3 Loader.

To query good data in S3 with Athena you can follow this guide: Using AWS Athena to query the 'good' bucket on S3

3

So ATOMIC.EVENTS is supposed to receive shredded events instead of enriched events?
How do I do that if I’m using Stream Enrich instead of EmrEtlRunner?

4

@Alexandre_Rayes, it depends on what ATOMIC.EVENTS you are talking about. When it comes to Redshift then yes, events should be stripped off those 3 fields as the data in them is loaded into separate tables.

To work with enriched data, we offer Snowplow Analytics SDK and you should take into consideration the enriched event’s structure as per the link I provided earlier.