We’re tremendously excited to release Snowplow RDB Loader 1.2.0. This release brings signficiant improvements into monitoring subsystem, including new webhook-based monitoring, reworked severity of the issues as well as ability to automigrate tables when length of the column has been increased.
Since we’ve released 1.0.0 many users brought up the suboptimal monitoring approach of the new RDB Loader - in particular it was quite easy to miss an error in
stdout that could require a human intervention in order to retry folder loading. With new webhook-based monitoring you can set RDB Loader to send all alarms onto a specified HTTP endpoint. All payloads will be schema’ed with
iglu:com.snowplowanalytics.monitoring.batch/alert/jsonschema/1-0-0, indicating what folder caused an alert and providing details necessary to resolve the problem.
Archive monitoring is our next step in improving monitoring subsystem. Previously, if Loader was down for some time, while S3DistCp and/or Shredder job were working users were running into cases where multiple folders were left unprocessed (in enriched archive) or unloaded (in shredded archive) and nothing was explicitly indicting it. It was possible to write custom monitoring script, but now this check exists in Loader itself. You can configure it to monitor folder with specified frequency to find out about data missing in Redshift.
Since R32, RDB Loader supports automigration - it can create tables and migrate them when a column was added, however another very common migration case has never been implemented - increase of
maxLength. Previously, user either still had to increase
VARCHAR columns manually to reflect the change or data was trimmed during loading.
For 1.2.0 Upgrade Guide please refer to our docs website: