I’ve seen the other threads about this error and I’ve tried a variety of versions for the Dataflow job at this point. I think I had the most success with 1.2.3 actually, but I am getting this error still.
Regardless, even without this issue, everything is ending up in the “bad” topic and never making it to good. I’m using GCP Dataflow and BigQuery with:
snowplow/beam-enrich:1.2.3 (but have tried latest and 1.1.0)
The “vanilla” iglu_resolver.json with
I am using the ip_lookups enrichment with MaxMind as well.
I’m new to setting this all up and everything seems to be running and collecting and putting messages into PubSub alright…It just won’t get past the Beam Enrich job. Messages end up in the bad topic, so I don’t even know if the loader will put things into BigQuery. Though the mutator created the initial schema alright.
I’ve read official documentation, tutorials, and searched the Scala source code for days trying to piece things together. I’ve tried a variety of configurations and am just at a loss…But sooo close, I can feel it. If I can figure it out, I’m more than happy to list everything out somewhere if it helps others I’m really excited about this tool and have been dying to use it for a while now.
Any suggestions or places to look? Thanks!