Hi @clin407,
have a sneaking suspicion it has to do with the bad enrichment data files
So just to clarify/make it easier to find help with this kind of thing - what you’re looking at here is bad rows. The Enrich component of Snowplow does two things - validate data against its event definition schema, and enriches the data.
If data doesn’t validate (ie doesn’t appear as expected), it goes to bad rows for debugging. The idea here is that you get data collection right - in other words, you ensure high quality data by making sure it’s collected correctly at source. That eliminates the need to spend a long time working on data quality/investigating problems later on - the data is usable once it lands in-DB.
- for this data, is it possible to change my sku so it accepts integers as opposed to strings?
Using the standard add to cart tracking no, you can’t change it. You could set it up as a custom event, but I think we’re missing the actual issue if we go down that road. SKUs aren’t integers, they’re essentially names of items - which are strings. They can have letters in them and it doesn’t make sense to add, subtract, multiply skus. So the simple solution is to just send the data as strings.
I recommend ensuring that you’ve tested a tracking setup thoroughly before taking it live, bad rows are generally for debugging issues with live tracking, but if you spin up a Snowplow Mini instance and test your tracking setup before implementing tracking, you’ll avoid this kind of thing by and large.
In terms of handling current tracking issues in production - here’s a guide to the process of deugging bad rows in Athena for batch (older versions) and here’s one for real-time (newer versions).
It’s definitely worth following that guide to uncover any other validation issues you might have.
What’s the best way to rerun all my bad enrichments? I’m assuming it will reprocess the bad data with the new configurations and therefore populate the add to cart and remove from cart tables? For my emr etl runner, I have something like this running:
If you want to reprocess the data, you need to convert the type of these values to strings.
It might be a bit of a time investment, but there’s a guide here to handling it.
Snowplow event recovery could be used to handle this
Or can i change it so it can accept both integers and strings?
It’s not possible for one field to have two types. Think of a column in a database - you can’t have strings in an integer column or vice versa.
I hope this is helpful.
Best,