Self-describing events without "real" events?

[dilyan]

@pranas You can use Athena to query the enriched events:

Using AWS Athena to query the 'good' bucket on S3

Since AWS launched Athena late last year, we’ve been using it to query the ‘bad’ data (i.e. data that fails validation) directly on S3. (Check out our tutorial on how to do that: Debugging bad rows in Athena.) Athena also holds great promise for querying “good” Snowplow data in S3, potentially as an alternative to, or alongside, querying it in Redshift. Athena is a particularly attractive querying tool for users who have very large volumes of Snowplow data, but have very variable query workload…

We also have a guide on using Athena to query the shredded events, but that comes with some limitations:

Using AWS Athena to query the shredded events

Recently, we looked into using AWS Athena to query the ‘good’ bucket on S3. The approach we outlined focussed on querying the ‘enriched’ unshredded data but we also wanted to see if we can query the shredded events directly from S3. Why use Athena? We’ve been using Athena to query the ‘bad’ data (ie data that fails validation) directly on S3. However, it also holds great promise for querying the ‘good’ Snowplow data in S3, potentially as an alternative to, or alongside, querying it in Redshift.…

EDIT:

You can use Athena to query the good bucket, which contains the enriched (but not yet shredded!) events.

Find your ‘orphan’ self-describing events in Redshift and see if their root_id matches any event_id in the enriched_events table in Athena.