More recently, the Beam SDKs have supported a templating method allowing dynamic runtime parameters to be passed directly to a Beam job.
I think it would be ‘neato’ if it were possible to compile to a Dataflow template, perhaps in addition to the existing Docker/binary options.
Although not a massive change, the advantages that I can see are:
- No requirement to initiate the pipeline via docker/pod or an environment specifically setup for initiating beam pipelines (jdk8 ) - e.g. managing jobs via CLI/gcloud directly rather than indirectly via a VM or container environment.
- Easier upgrade path for users/ease of modification of parameters of an older pipeline.
- Resource files for the job are pre-staged, so faster startup/shutdown times.
Let me know what you think.