So I have managed to fully orchestrate all relevant docker containers for a native GCP setup onto a Kubernetes cluster, where both the BQ loader and the beam enrich each fire up a dataflow job. All containers are run using a deployment.yaml each.
After firing up the containers, the beam-enrich workload keeps sending out errors and failing the deployment (see picture below)
The logs show this
Why are they keep failing? According to the logs, it seems like an log messages are classified as “errors” even though they only are “info”. Can this be the source of the problem? This is quite bothersome if you would change this to an --update job instead. Would it mean that the deployment would fail all the time hence updating the dataflow job even though nothing new has been added?
Any advice on this? Thanks!