Nice work on the Spark release! Our pipeline ran successfully a few times, but as I was experimenting with instance types, the Shred step failed 2 hours into the job. This is probably memory-related, but I wasn’t expecting this with
4x c4.4xlarge instances (each with 30GB of memory).
Here’s the stderr file from one of the containers: