S3 Loader 0.6.0 - unable to define Directory Pattern

1

I notice that a recent commit to the s3 loader introduced functionality supporting custom directory patterns, namely for a YYYY/MM/DD format, however I am unable to get this working and instead it seems that the directory pattern is ignored all together.

My code for the s3 portion of the config is below.

s3 {
  region = "us-east-1"
  bucket = "xx-snowplow-logs/sp-s3-sink"
  directoryPattern = "good/{YYYY}/{MM}/{DD}"
  # optional directory structure to use while storing data on s3, you can place date format strings which will be
  # replaced with date time values
  # eg: directoryPattern = "enriched/good/run_yr={YYYY}/run_mo={MM}/run_dy={DD}"
  # Format is one of lzo or gzip
  # Note, that you can use gzip only for enriched data stream.
  format = "gzip"

  # Maximum Timeout that the application is allowed to fail for (in milliseconds)
  maxTimeout = 120000000

  ## Optional endpoint url configuration to override aws s3 endpoints,
  ## this can be used to specify local endpoints when using localstack
  # customEndpoint = {{kinesisEndpoint}}
}

the result is a single folder xx-snowplow-logs/sp-s3-sink without any partitioning.

I would appreciate any assistance.

Thankyou

2

Hey @blanche - I’m running several instances of the 0.7.0 s3 loader, with directoryPattern configured as yours above.

I did a little poking around in github, and it appears that directoryPattern went out with the 0.7.0-rc1 release. Running that version should solve the problem you’re seeing.

Hope this helps.

Jake

2 Likes
3

Makes sense - cheers for the response @jakethomas