BQ Stream Loader on GCP AppEngine

For engineers GCP pipeline
1

Hi,

I am trying to deploy the bq stream loader 1.0.0 jar on appengine service.

The deployment is a success when my dockerfile looks like this (the config and resolver files are in base64 format)

FROM openjdk:18-jdk-alpine

COPY snowplow-bigquery-streamloader-1.0.0.jar snowplow-bigquery-streamloader-1.0.0.jar
COPY config.hocon config.hocon
COPY resolver.json resolver.json

CMD ["java","-jar","snowplow-bigquery-streamloader-1.0.0.jar","--config","ewogICJwcm9qZW.........","--resolver","ewogICJzY2hlb........"]

But the deployment fails with an illegal character exception when the same file looks like this, (config and resolver are being encoded on the go or when using env variables)

On the go conversion:

FROM openjdk:18-jdk-alpine

COPY snowplow-bigquery-streamloader-1.0.0.jar snowplow-bigquery-streamloader-1.0.0.jar
COPY config.hocon config.hocon
COPY resolver.json resolver.json

# ENV CONFIG=
# ENV RESOLVER=

CMD ["java","-jar","snowplow-bigquery-streamloader-1.0.0.jar","--config","$(cat config.hocon | base64 -w 0)","--resolver","$(cat resolver.json | base64 -w 0)"]

# CMD ["java","-jar","snowplow-bigquery-streamloader-1.0.0.jar","--config","$CONFIG","--resolver","$RESOLVER"]

Error:
image

Could someone help out?

Thank you!

1 Like
2

I have the same problem, I am trying to setup the snowplow BigQuery services and have issues with illegal characters. Has anyone in the community successfully setup these using Google Cloud Platform? If so how.

3

Hi @amar-p6 ,

You can write the command in script file and run the script file in docker.

Hope that helped