SQL Runner running multiple copies of playbook?

For data modelers & consumers SQL Runner
1

I’m currently troubleshooting a problem with a SQL-based ETL running on Redshift with SQL Runner 0.5.2.

With the help of AWS Support, I found that it appears the SQL Runner playbook is being run twice; once when it’s kicked off by our cron-based shell script, and then again a few hours later. The SQL files conflict with each other, causing unpredictable results.

I’ve just upgraded to version 0.6.0 and enabled lockfiles to see if that resolves the issue

Has anyone seen this behaviour before?

2

Hi @iain, that sounds very odd. Have you checked:

  • That your crontab only specifies the first run (and not the second)
  • That you don’t have a separate process (e.g. a scheduler in your dev environment) also running?
3

Hi @alex,

Thanks, I checked those. The jobs were all created with the same connection according to the Redshift connection log. It looks like the connection was opened when the first job started and closed when the second one finished.

The problem hasn’t recurred since I set up locks and migrated to new version. I’ll keep an eye out though.