Hi, I have setup a snowplow pipeline in GCP. I am able to load events table in BigQuery without any issue until I tried the below.
I tried to setup a similar pipeline where my target table is a time partitioned table. I’ve read the discussion Google Cloud Platform data pipeline optimization. I created the partitioned table based on the derived_tstamp column (with DAY granularity) manually before running the pipeline with the same schema suggested in atomic schema. But, as I am firing events from the tracker, the events are not loading to the table rather moving to the failed inserts topic in PubSub although the mutator is able to mutate the table and custom columns are getting added to the table. Could you please suggest what is the possible reason and how to resolve?
Also, my ultimate aim is to create a partitioned table with event_name as the partitioned column. Please advise how can I achieve the same.