Archive shred step is not working


#1

Our EMR jobs are not starting since 2-3 days. When I manually started a job it is showing there are contents in shredded good so I ran job using –resume-from archive_shredded but it is actually not working, though EMR job shows completed there are still files in shredded good bucket


#2

@Dev what version are you on? i had the same problem. upgrading will fix it. i’m assuming it finished all steps previously and when you try to run a new full run it fails because of that error above. the problem could be because there are too many $folder in your directory.

this script helps too.

https://discourse.snowplowanalytics.com/t/enrich-good-folder-contains-empty-run-date-folder-files/1737/13