I had created a custom docker image for the enrich pubsub (though there is a docker image provided), the dockerfile for the same is as follows:
FROM openjdk:12 COPY /enrichments /enrichments COPY config.hocon config.hocon COPY resolver.json resolver.json COPY snowplow-enrich-pubsub-2.0.3.jar snowplow-enrich-pubsub-2.0.3.jar COPY script.sh script.sh RUN yum install jq -y CMD sh script.sh
Contents of script.sh being:
jq '.data.repositories.connection.http.uri=env.SCHEMA_BUCKET' resolver.json >> tmp.json && mv tmp.json resolver.json java -jar snowplow-enrich-pubsub-2.0.3.jar --enrichments enrichments --iglu-config resolver.json --config config.hocon
I was able to successfully deploy it on AppEngine Flexible and have sent a load of 12 millions records to the same. The enricher scaled to a max of 20 instances and still wasn’t able to clear the backlog even after a few hours.
Just wanted to know if I’m missing something