How is it possible to change the dataflow job name of the BigQuery loader 0.4.0? I’ve used the same syntax to run my beam-enrich pipeline but it does not work, while my beam-enrich job name does work. This is important to me to separate my development and production pipeline.
Hi @sdbeuf, apologies, I misunderstood your question.
As well as labels that you can pass to the Dataflow job, you can also pass the --jobName=myJobName parameter when launching the job. This will create a job called myJobName.
As for the links you quote, the second one is the schema describing what the config file should look like, and the snippet in the docs is an example of a config that complies with that schema.
That did the trick thanks, for some reason in the beam-enrich --job-name works but in the loader not. Changed it in the loader --jobName and that works.
For Python the job name label is job-name while Java it is jobName. I guess the bq-loader is implemented using the Java SDK which would make sense since the loader is a Scala-based application. https://github.com/snowplow-incubator/snowplow-bigquery-loader