A few hours ago our Enrich suddenly started diverting 100% of its input to its bad stream, tagging each record with:
"error: Unexpected exception fetching iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0 in HTTP Iglu repository IgluCentral: java.io.IOException: Server returned HTTP response code: 500 for URL: http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0\n level: \"error\"\n"
Enrich continued rejecting all of its input with this same message until an hour later when we spotted the problem and restarted it. We’re resubmitting those bad stream records to Enrich now, and it is accepting them this time.
Any suggestions to what might have caused this, and maybe how we could prevent its recurrence?
(in case it matters, we’re running Enrich 0.10.0. And yes I realize that this is kind of old and we need to upgrade it someday…)
Editing to add: I should also have mentioned that I looked in my /var/log/enrich.log for anything potentially related, but didn’t find anything.