Snowplow rc release not working as expected


#1

With snowplow_emr_r88_angkor_wat_rc4.zip
or any other angkor_wat released I am facing following issues:

./snowplow-storage-loader --config exe-snowplow/config/config.yml --targets /root/exe-snowplow/data/ --resolver exe-snowplow/config/resolver.json 

It just archive events but does not download to the target.

Initial reponse on running storage loader

Archiving Snowplow events...
  moving files from s3://xxx-bucket/enriched/good/ to s3://xxx-bucket/enriched/archive/
(t0)    MOVE xxx-bucket/enriched/good/run=2017-04-03-20-01-14/part-00000 -> xxx-bucket/enriched/archive/run=2017-04-03-20-01-14/part-00000
(t1)    MOVE xxx-bucket/enriched/good/run=2017-04-03-20-01-14/part-00001 -> xxx-bucket/enriched/archive/run=2017-04-03-20-01-14/part-00001(t2)    MOVE xxx-bucket/enriched/good/run=2017-04-03-20-01-14/part-00002 -> xxx-bucket/enriched/archive/run=2017-04-03-20-01-14/part-00002(t3)    MOVE xxx-bucket/enriched/good/run=2017-04-03-20-01-14/part-00003 -> xxx-bucket/enriched/archive/run=2017-04-03-20-01-14/part-00003(t4)    MOVE xxx-bucket/enriched/good/run=2017-04-03-20-01-14/part-00004 -> xxx-bucket/enriched/archive/run=2017-04-03-20-01-14/part-00004



      +-> xxx-bucket/enriched/archive/run=2017-04-03-20-01-14/part-00004      +-> xxx-bucket/enriched/archive/run=2017-04-03-20-01-14/part-00001

      +-> xxx-bucket/enriched/archive/run=2017-04-03-20-01-14/part-00003
      +-> xxx-bucket/enriched/archive/run=2017-04-03-20-01-14/part-00000
      +-> xxx-bucket/enriched/archive/run=2017-04-03-20-01-14/part-00002
      x xxx-bucket/enriched/good/run=2017-04-03-20-01-14/part-00000      x xxx-bucket/enriched/good/run=2017-04-03-20-01-14/part-00004

      x xxx-bucket/enriched/good/run=2017-04-03-20-01-14/part-00002
      x xxx-bucket/enriched/good/run=2017-04-03-20-01-14/part-00001      x xxx-bucket/enriched/good/run=2017-04-03-20-01-14/part-00003

  moving files from s3://xxx-bucket/enriched/good/ to s3://xxx-bucket/enriched/archive/
(t0)    MOVE xxx-bucket/enriched/good/run=2017-04-03-20-01-14/_SUCCESS -> xxx-bucket/enriched/archive/run=2017-04-03-20-01-14/_SUCCESS
      +-> xxx-bucket/enriched/archive/run=2017-04-03-20-01-14/_SUCCESS
      x xxx-bucket/enriched/good/run=2017-04-03-20-01-14/_SUCCESS
  moving files from s3://xxx-bucket/shredded/good/ to s3://xxx-bucket/shredded/archive/
(t1)    MOVE xxx-bucket/shredded/good/run=2017-04-03-20-01-14/atomic-events/part-00001 -> xxx-bucket/shredded/archive/run=2017-04-03-20-01-14/atomic-events/part-00001(t2)    MOVE xxx-bucket/shredded/good/run=2017-04-03-20-01-14/atomic-events/part-00002 -> xxx-bucket/shredded/archive/run=2017-04-03-20-01-14/atomic-events/part-00002(t4)    MOVE xxx-bucket/shredded/good/run=2017-04-03-20-01-14/atomic-events/part-00003 -> xxx-bucket/shredded/archive/run=2017-04-03-20-01-14/atomic-events/part-00003


(t0)    MOVE xxx-bucket/shredded/good/run=2017-04-03-20-01-14/atomic-events/part-00000 -> xxx-bucket/shredded/archive/run=2017-04-03-20-01-14/atomic-events/part-00000
      +-> xxx-bucket/shredded/archive/run=2017-04-03-20-01-14/atomic-events/part-00001
      +-> xxx-bucket/shredded/archive/run=2017-04-03-20-01-14/atomic-events/part-00003
      +-> xxx-bucket/shredded/archive/run=2017-04-03-20-01-14/atomic-events/part-00002      +-> xxx-bucket/shredded/archive/run=2017-04-03-20-01-14/atomic-events/part-00000

      x xxx-bucket/shredded/good/run=2017-04-03-20-01-14/atomic-events/part-00001
      x xxx-bucket/shredded/good/run=2017-04-03-20-01-14/atomic-events/part-00000
      x xxx-bucket/shredded/good/run=2017-04-03-20-01-14/atomic-events/part-00003
      x xxx-bucket/shredded/good/run=2017-04-03-20-01-14/atomic-events/part-00002
  moving files from s3://xxx-bucket/shredded/good/ to s3://xxx-bucket/shredded/archive/
Completed successfully

Response on later runs

Archiving Snowplow events...
  moving files from s3://xxx-bucket/enriched/good/ to s3://xxx-bucket/enriched/archive/
  moving files from s3://xxx-bucket/enriched/good/ to s3://xxx-bucket/enriched/archive/
  moving files from s3://xxx-bucket/shredded/good/ to s3://xxx-bucket/shredded/archive/
  moving files from s3://xxx-bucket/shredded/good/ to s3://xxx-bucket/shredded/archive/
Completed successfully

Download does not initiate.

kindly help


#2

Hi @v3nom,

You are attempting to use a Snowplow release candidate (“rc”) - Snowplow rcs are designed for internal testing and often have significant changes relative to previous final releases. We don’t update the Snowplow documentation until the rc process has been finalized, so effectively you are using one version of Snowplow with documentation and example config files from another version.

The specific problem you are experiencing is that storage targets for StorageLoader in R88 are configured in a different way to R87.

To resolve this, simply use R87.


No data loaded in postgres, no errors either
#3

@alex

I have tried all other packages before raising this issue. The issue occurred while using R87

Package: snowplow_emr_r87_chichen_itza.zip

Downloading Snowplow events...
Unexpected error: undefined method `host=' for #<Fog::Storage::AWS::Real:0x48a61e38>
uri:classloader:/storage-loader/lib/snowplow-storage-loader/s3_tasks.rb:41:in `download_events'
uri:classloader:/storage-loader/bin/snowplow-storage-loader:42:in `<main>'
org/jruby/RubyKernel.java:973:in `load'
uri:classloader:/META-INF/main.rb:1:in `<main>'
org/jruby/RubyKernel.java:955:in `require'
uri:classloader:/META-INF/main.rb:1:in `(root)'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1:in `<main>'

check: Storage-loader issues, runtime errors

here @anton has mentioned the resolution for same in the rc release but I don’t know why it deos not load other methods except archive.