I am setting up latest Snowplow ETL Batch processing. (R119)
ETL process is getting completed successfully. During review process found out that there are lots of zero 0 byte files generated under Shredded Bad bucket.
Want to know whats the significance of these files and why these are getting created?
Is there any configuration issue because of which proper files are not getting created. ?
Hi @sp_user,
It is normal, the shred process relies on some underlying AWS tools that can sometimes create empty files in the bucket.
I might be wrong here but I think it’s related to S3DistCP.
In any case - it’s not a sign of a problem, however when you have a large amount of these files I think it can slow down the ETL process, so it’s best to clean them out of the bucket every once in a while.