Launch Beam Enrich instance group inside subnet(GCP)

For engineers Enrichment
1

Hi Mr or Ms,
Currently we are running beam-enrich fine on GCP but there is issue that we want the dataflow beam enrich instance group to run inside our subnet:

image
image453×951 51.5 KB

As you can see, there is no subnet selected right now.

And below is the script we put in instance template:

sudo docker run \
  -v $PWD/snowplow/config:/snowplow/config \
  snowplow/beam-enrich:$enrich_version \
  --runner=DataFlowRunner \
  --project=$project_id \
  --streaming=true \
  --zone=$zone \
  --gcpTempLocation=gs://$temp_bucket/temp-files/ \
  --job-name=beam-enrich \
  --raw=projects/$project_id/subscriptions/good-sub \
  --enriched=projects/$project_id/topics/enriched-good \
  --bad=projects/$project_id/topics/enriched-bad \
  --resolver=/snowplow/config/iglu_resolver.json \
  --enrichments=/snowplow/config/enrichments/

Is there a way to put an option in the script to make it happen? Thank you

2

Have a look at the ‘Security and Networking’ options for Dataflow here. The Docker container should pass these pipeline options through.

3

Thank you Mike, we tried using the subnetwork option, but GCP returns us an error says the subnetwork does not exist.
The option format we put in is something like this:
subnetwork=https://www.googleapis.com/compute/v1/projects/{project-name}/regions/$region/subnetworks/{subnetwork-name}

Please let me know if you have experience in this, I guess it is probably this option not put in the right format

4

That’s the right format - if it’s coming back with that error I’d double check permissions as well as that the subnetwork exists in the same zone that you’re attempting to launch the Dataflow workers in.