Enriched HDFS -> S3 stuck forever

For engineers Enrichment
1

Hi I’m stuck with Enriched HDFS → S3 step, where logs says it’s still running and forever. no other logs/error or nothing kind of clueless.

Appreciate if any can help.

have 5 steps

→ Elasticity Scalding Step: Enrich Raw Events
→ Elasticity S3DistCp Step: Enriched HDFS → S3 (Stuck forever here)
→ Elasticity S3DistCp Step: Enriched HDFS _SUCCESS → S3
→ Elasticity Scalding Step: Shred Enriched Events
→ Elasticity S3DistCp Step: Shredded HDFS → S3

This whole pipeline is use to run in 30min. and today it’s been more than 15 hours.

snowplow version - r75-long-legged-buzzard

Surprise is, I’ve 3 pipelines of same configs, 1 of them is working perfectly fine in ireland region, but 2 of them from singapore region are stuck and running forever

2

It sounds like it could have been a region-specific issue. Were you able to resolve this?

3

This turns out to be, infrastructure issue, I just upgrade instance type to higher version keeping same number of machines counts. Solved the problem