Trying to upgrade to R32

1

As mentioned in the documentation here - https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/loaders-storage-targets/snowplow-rdb-loader/r32-upgrade-guide/
I have change the config.yaml to use as follwing:

aws:
  emr:
    ami_version: 5.19.0     # was 5.9.0; Required by RDB Shredder 
storage:
  versions:
    rdb_loader: 0.17.0      # was 0.16.0
    rdb_shredder: 0.16.0    # was 0.15.0

However, when I change the json schema version for in redshift.json to 4-0-0, I am getting following error -

ubuntu@ip-10-0-1-121:~/rr-snowplow/upgrade/modules/emr_etl_runner$ ./snowplow-emr-etl-runner-r119 run -c rr_emr_etl_stream_upgrade.yml -r resolver.json -t targets/$RR_SNOWPLOW_APP-$RR_SNOWPLOW_ENV
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/tmp/jruby-15987/jruby8941945816020454867jopenssl.jar) to constructor java.security.cert.CertificateFactory(java.security.cert.CertificateFactorySpi,java.security.Provider,java.lang.String)
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
uri:classloader:/gems/avro-1.8.1/lib/avro/schema.rb:350: warning: constant ::Fixnum is deprecated
uri:classloader:/gems/json-schema-2.7.0/lib/json-schema/util/array_set.rb:18: warning: constant ::Fixnum is deprecated
Error in [redshift.json] The property '#/' contains additional properties ["sslMode"] outside of the schema when none are allowed
UncaughtThrowError: uncaught throw #<JSON::Schema::ValidationError: The property '#/' contains additional properties ["sslMode"] outside of the schema when none are allowed in schema 4c567ae7-701b-5d73-9d27-26d4bfb9923d>
                      throw at org/jruby/RubyKernel.java:1203
  block in validate_targets at uri:classloader:/emr-etl-runner/lib/snowplow-emr-etl-runner/runner.rb:202
                        map at org/jruby/RubyArray.java:2557
           validate_targets at uri:classloader:/emr-etl-runner/lib/snowplow-emr-etl-runner/runner.rb:193
                    send_to at uri:classloader:/gems/contracts-0.11.0/lib/contracts/method_reference.rb:43
                  call_with at uri:classloader:/gems/contracts-0.11.0/lib/contracts/call_with.rb:76
   block in redefine_method at uri:classloader:/gems/contracts-0.11.0/lib/contracts/method_handler.rb:138
                 initialize at uri:classloader:/emr-etl-runner/lib/snowplow-emr-etl-runner/runner.rb:99
                    send_to at uri:classloader:/gems/contracts-0.11.0/lib/contracts/method_reference.rb:43
                  call_with at uri:classloader:/gems/contracts-0.11.0/lib/contracts/call_with.rb:76
   block in redefine_method at uri:classloader:/gems/contracts-0.11.0/lib/contracts/method_handler.rb:138
                     <main> at uri:classloader:/emr-etl-runner/bin/snowplow-emr-etl-runner:40
                       load at org/jruby/RubyKernel.java:994
                     <main> at uri:classloader:/META-INF/main.rb:1
                    require at org/jruby/RubyKernel.java:970
                     (root) at uri:classloader:/META-INF/main.rb:1
                     <main> at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
ERROR: org.jruby.embed.EvalFailedException: (UncaughtThrowError) uncaught throw #<JSON::Schema::ValidationError: The property '#/' contains additional properties ["sslMode"] outside of the schema when none are allowed in schema 4c567ae7-701b-5d73-9d27-26d4bfb9923d>
2

Hi @Tejas_Behra ,

We encourage you to upgrade to the latest version of RDB loader, 1.1.0. You can find the upgrade guide here.

Could you share the content of your config files please ? Based on the logs there seems to be a problem with sslMode.

3 
{
    "schema": "iglu:com.snowplowanalytics.snowplow.storage/redshift_config/jsonschema/2-1-0",
    "data": {
        "name": "AWS Redshift enriched events storage",
        "host": "e2-analytics-dev.cjr88dydsm3q.us-east-1.redshift.amazonaws.com",
        "database": "snowplow",
        "port": 5439,
        "sslMode": "REQUIRE",
        "username": "mpadmin",
        "password": "xxxxxxxxxxxxxxx",
        "roleArn": "arn:aws:iam::693704117329:role/redshift-s3-full",
        "schema": "atomic",
        "maxError": 1,
        "compRows": 20000,
        "sshTunnel": null,
        "purpose": "ENRICHED_EVENTS"
    }
}
~
4

Hi @BenB, I would like to upgrade to RDB Loader to 1.1.0 but could not find enough resources (config & playbook example for dataflow runner). Also, I am assuming that upgrading to 1.1.0 directly is not possible and I have to go through R33, R34, R35, 1.0.0 & then 1.1.0. Please correct me if I am wrong

5

Hi @Tejas_Behra,

They can be found on our docs website.

You can upgrade directly to 1.1.0, following this guide. Please note that now Redshift tables for contexts are automatically created by the loader, if you shred data as TSV.

This is not correct. Only verify-ca and verify-full are possible values. You can find the schema definition here.

1 Like