ETL Shred is consistently failing


#1

… Continuing the discussion from ETL Shred step taking longer and longer:

Didn’t want to hijack the thread linked above. So, here’s a new one. My pipeline is dead in the water. Error logs are cryptic - and make very little sense to me. What I observe is enrich is finishing rather quickly, shred takes abnormally longer. At some point it stalls, drops a few core nodes, resizes and then exits with errors. Console screenshot attached, maybe a massive coincidence, but it always breaks in the same place.

Errors I see are ( I sampled multiple repeated lines )

2017-03-28 04:37:45,247 WARN org.apache.hadoop.hdfs.DFSClient (IPC Server handler 2 on 10020): Failed to connect to /172.30.0.144:50010 for block, add to deadNodes and continue. java.net.NoRouteToHostException: No route to host
2017-03-28 03:43:16,271 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl (AsyncDispatcher event handler): Updating application attempt appattempt_1490669380627_0008_000001 with final state: FAILED, and exit status: -100
2017-03-28 03:43:16,272 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl (AsyncDispatcher event handler): appattempt_1490669380627_0008_000001 State change from FINAL_SAVING to FAILED
2017-03-28 03:43:16,272 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl (AsyncDispatcher event handler): The number of failed attempts is 0. The max attempts is 2
2017-03-28 04:35:14,286 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger (AsyncDispatcher event handler): USER=hadoop   OPERATION=Application Finished - Failed TARGET=RMAppManager     RESULT=FAILURE  DESCRIPTION=App failed with state: FAILED       PERMISSIONS=Application application_1490669380627_0009 failed 2 times due to AM Container for appattempt_1490669380627_0009_000003 exited with  exitCode: -1000
Failing this attempt. Failing the application.  APPID=application_1490669380627_0009

I doubled the compute nodes, added more juice to the master node ( it seemed to me the memory and disk capacity were creeping dangerously close to the redline ), didn’t make a slightest dent.

I’m at my wit’s end with this one. Every attempt to re-run produces different class of errors. Some are indicative of master node loosing its mind ( HDFS blocks missing ) some are as cryptic as the samples above.

  • Any ideas?
  • I’ve tried to use scala common enrich/shred as a dependency for for a realtime kinesis(Enriched)->[hypothetical service]->s3 (shredded)->redshift last resort development effort, but I can’t figure out how to use the library in a Java context. My scala authoring skills are non-existent Has anyone managed to develop streaming shredder?
  • Any pointers to setting up a spark beta pipeline?
  • We’ve recently added a few custom unstructured events, but only tests made it into the pipeline, no significant volume to speak of. I’ve checked the assets ( jsonpaths ) are in the right place on s3 and schemas are happily congregating in Iglu scala server. Maybe I have missed something, new event-wise?

#2
2017-03-28 07:00:19,974 INFO org.apache.hadoop.mapred.ClientServiceDelegate (flow com.snowplowanalytics.snowplow.enrich.hadoop.ShredJob): Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server    
2017-03-28 07:00:51,623 WARN cascading.flow.FlowStep (flow com.snowplowanalytics.snowplow.enrich.hadoop.ShredJob): [com.snowplowanalytics....] unable to kill job: (3/7)
java.io.IOException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.YarnRuntimeException): Could not load history file hdfs://ip-172-30-25-193.ec2.internal:8020/tmp/hadoop-yarn/staging/history/done/2017/03/28/000000/job_1490677498829_0008-1490679714162-hadoop-%5BABC39635C9CD4B6DB5717397E3E4C9CA%2F9A0A9E5C683F4505-1490680143365-185-7-SUCCEEDED-default-1490679718905.jhist
<...>

Has anyone had any luck with golang Job flow manager for the raw pipeline processing? Should I be switching?


#3

switched to dataflow-runner, increased resources from 4 core nodes to 5 core + 5 task

  • distributed enriched events from s3 to hdfs
  • distributed raw events from s3 to hdfs (just in case)
  • ran the shred job with the same parameters as the step was configured on the failed runs

same exact behavior - runs almost to the end, then fails the shred step. over and over again. Suggestions? What logs could be indicative of the root cause?

BTW, hdfs /local/snowplow/shreded-events is never created

  • Input size is at most 10 Gigs - how can it blow through 200G cluster capacity?

$ hdfs dfs -ls  /local/snowplow/*
Found 150 items
-rw-r--r--   2 hadoop hadoop          0 2017-03-28 12:01 /local/snowplow/enriched-events/_SUCCESS
-rw-r--r--   2 hadoop hadoop   67732746 2017-03-28 12:01 /local/snowplow/enriched-events/part-00000
-rw-r--r--   2 hadoop hadoop   65903010 2017-03-28 12:01 /local/snowplow/enriched-events/part-00001
-rw-r--r--   2 hadoop hadoop   63776876 2017-03-28 12:01 /local/snowplow/enriched-events/part-00002
-rw-r--r--   2 hadoop hadoop   63058667 2017-03-28 12:01 /local/snowplow/enriched-events/part-00003
-rw-r--r--   2 hadoop hadoop   62819144 2017-03-28 12:01 /local/snowplow/enriched-events/part-00004
-rw-r--r--   2 hadoop hadoop   62823914 2017-03-28 12:01 /local/snowplow/enriched-events/part-00005
-rw-r--r--   2 hadoop hadoop   62859731 2017-03-28 12:01 /local/snowplow/enriched-events/part-00006
-rw-r--r--   2 hadoop hadoop   62889238 2017-03-28 12:01 /local/snowplow/enriched-events/part-00007
-rw-r--r--   2 hadoop hadoop   62842838 2017-03-28 12:01 /local/snowplow/enriched-events/part-00008
-rw-r--r--   2 hadoop hadoop   62911028 2017-03-28 12:01 /local/snowplow/enriched-events/part-00009
-rw-r--r--   2 hadoop hadoop   62871132 2017-03-28 12:01 /local/snowplow/enriched-events/part-00010
-rw-r--r--   2 hadoop hadoop   62864665 2017-03-28 12:01 /local/snowplow/enriched-events/part-00011
-rw-r--r--   2 hadoop hadoop   62856645 2017-03-28 12:01 /local/snowplow/enriched-events/part-00012
-rw-r--r--   2 hadoop hadoop   62831377 2017-03-28 12:01 /local/snowplow/enriched-events/part-00013
-rw-r--r--   2 hadoop hadoop   62846962 2017-03-28 12:01 /local/snowplow/enriched-events/part-00014
-rw-r--r--   2 hadoop hadoop   62871229 2017-03-28 12:01 /local/snowplow/enriched-events/part-00015
-rw-r--r--   2 hadoop hadoop   62865110 2017-03-28 12:01 /local/snowplow/enriched-events/part-00016
-rw-r--r--   2 hadoop hadoop   62859158 2017-03-28 12:01 /local/snowplow/enriched-events/part-00017
-rw-r--r--   2 hadoop hadoop   62902994 2017-03-28 12:01 /local/snowplow/enriched-events/part-00018
-rw-r--r--   2 hadoop hadoop   62862098 2017-03-28 12:01 /local/snowplow/enriched-events/part-00019
-rw-r--r--   2 hadoop hadoop   62881219 2017-03-28 12:01 /local/snowplow/enriched-events/part-00020
-rw-r--r--   2 hadoop hadoop   62886105 2017-03-28 12:01 /local/snowplow/enriched-events/part-00021
-rw-r--r--   2 hadoop hadoop   62904136 2017-03-28 12:01 /local/snowplow/enriched-events/part-00022
-rw-r--r--   2 hadoop hadoop   62884892 2017-03-28 12:01 /local/snowplow/enriched-events/part-00023
-rw-r--r--   2 hadoop hadoop   62877433 2017-03-28 12:01 /local/snowplow/enriched-events/part-00024
-rw-r--r--   2 hadoop hadoop   62893068 2017-03-28 12:01 /local/snowplow/enriched-events/part-00025
-rw-r--r--   2 hadoop hadoop   62906402 2017-03-28 12:01 /local/snowplow/enriched-events/part-00026
-rw-r--r--   2 hadoop hadoop   62877341 2017-03-28 12:01 /local/snowplow/enriched-events/part-00027
-rw-r--r--   2 hadoop hadoop   62842047 2017-03-28 12:01 /local/snowplow/enriched-events/part-00028
-rw-r--r--   2 hadoop hadoop   62804312 2017-03-28 12:01 /local/snowplow/enriched-events/part-00029
-rw-r--r--   2 hadoop hadoop   62873489 2017-03-28 12:01 /local/snowplow/enriched-events/part-00030
-rw-r--r--   2 hadoop hadoop   62833494 2017-03-28 12:01 /local/snowplow/enriched-events/part-00031
-rw-r--r--   2 hadoop hadoop   62864866 2017-03-28 12:01 /local/snowplow/enriched-events/part-00032
-rw-r--r--   2 hadoop hadoop   62830798 2017-03-28 12:01 /local/snowplow/enriched-events/part-00033
-rw-r--r--   2 hadoop hadoop   62294320 2017-03-28 12:01 /local/snowplow/enriched-events/part-00034
-rw-r--r--   2 hadoop hadoop   62023931 2017-03-28 12:01 /local/snowplow/enriched-events/part-00035
-rw-r--r--   2 hadoop hadoop   59474608 2017-03-28 12:01 /local/snowplow/enriched-events/part-00036
-rw-r--r--   2 hadoop hadoop   59454627 2017-03-28 12:01 /local/snowplow/enriched-events/part-00037
-rw-r--r--   2 hadoop hadoop   58977620 2017-03-28 12:01 /local/snowplow/enriched-events/part-00038
-rw-r--r--   2 hadoop hadoop   58805121 2017-03-28 12:01 /local/snowplow/enriched-events/part-00039
-rw-r--r--   2 hadoop hadoop   58331440 2017-03-28 12:01 /local/snowplow/enriched-events/part-00040
-rw-r--r--   2 hadoop hadoop   57214946 2017-03-28 12:01 /local/snowplow/enriched-events/part-00041
-rw-r--r--   2 hadoop hadoop   57091913 2017-03-28 12:01 /local/snowplow/enriched-events/part-00042
-rw-r--r--   2 hadoop hadoop   56796956 2017-03-28 12:01 /local/snowplow/enriched-events/part-00043
-rw-r--r--   2 hadoop hadoop   56551823 2017-03-28 12:01 /local/snowplow/enriched-events/part-00044
-rw-r--r--   2 hadoop hadoop   56550024 2017-03-28 12:01 /local/snowplow/enriched-events/part-00045
-rw-r--r--   2 hadoop hadoop   55699664 2017-03-28 12:01 /local/snowplow/enriched-events/part-00046
-rw-r--r--   2 hadoop hadoop   55556318 2017-03-28 12:01 /local/snowplow/enriched-events/part-00047
-rw-r--r--   2 hadoop hadoop   55452747 2017-03-28 12:01 /local/snowplow/enriched-events/part-00048
-rw-r--r--   2 hadoop hadoop   55302671 2017-03-28 12:01 /local/snowplow/enriched-events/part-00049
-rw-r--r--   2 hadoop hadoop   55056696 2017-03-28 12:01 /local/snowplow/enriched-events/part-00050
-rw-r--r--   2 hadoop hadoop   54994053 2017-03-28 12:01 /local/snowplow/enriched-events/part-00051
-rw-r--r--   2 hadoop hadoop   54615231 2017-03-28 12:01 /local/snowplow/enriched-events/part-00052
-rw-r--r--   2 hadoop hadoop   54082577 2017-03-28 12:01 /local/snowplow/enriched-events/part-00053
-rw-r--r--   2 hadoop hadoop   53393179 2017-03-28 12:01 /local/snowplow/enriched-events/part-00054
-rw-r--r--   2 hadoop hadoop   53228624 2017-03-28 12:01 /local/snowplow/enriched-events/part-00055
-rw-r--r--   2 hadoop hadoop   53244768 2017-03-28 12:01 /local/snowplow/enriched-events/part-00056
-rw-r--r--   2 hadoop hadoop   53080196 2017-03-28 12:01 /local/snowplow/enriched-events/part-00057
-rw-r--r--   2 hadoop hadoop   52410421 2017-03-28 12:01 /local/snowplow/enriched-events/part-00058
-rw-r--r--   2 hadoop hadoop   52088536 2017-03-28 12:01 /local/snowplow/enriched-events/part-00059
-rw-r--r--   2 hadoop hadoop   50431824 2017-03-28 12:01 /local/snowplow/enriched-events/part-00060
-rw-r--r--   2 hadoop hadoop   50479304 2017-03-28 12:01 /local/snowplow/enriched-events/part-00061
-rw-r--r--   2 hadoop hadoop   49147948 2017-03-28 12:01 /local/snowplow/enriched-events/part-00062
-rw-r--r--   2 hadoop hadoop   48982423 2017-03-28 12:01 /local/snowplow/enriched-events/part-00063
-rw-r--r--   2 hadoop hadoop   48968406 2017-03-28 12:01 /local/snowplow/enriched-events/part-00064
-rw-r--r--   2 hadoop hadoop   47719325 2017-03-28 12:01 /local/snowplow/enriched-events/part-00065
-rw-r--r--   2 hadoop hadoop   46280694 2017-03-28 12:01 /local/snowplow/enriched-events/part-00066
-rw-r--r--   2 hadoop hadoop   46022580 2017-03-28 12:01 /local/snowplow/enriched-events/part-00067
-rw-r--r--   2 hadoop hadoop   46015069 2017-03-28 12:01 /local/snowplow/enriched-events/part-00068
-rw-r--r--   2 hadoop hadoop   45938216 2017-03-28 12:01 /local/snowplow/enriched-events/part-00069
-rw-r--r--   2 hadoop hadoop   45686473 2017-03-28 12:01 /local/snowplow/enriched-events/part-00070
-rw-r--r--   2 hadoop hadoop   45519307 2017-03-28 12:01 /local/snowplow/enriched-events/part-00071
-rw-r--r--   2 hadoop hadoop   45229533 2017-03-28 12:01 /local/snowplow/enriched-events/part-00072
-rw-r--r--   2 hadoop hadoop   45099189 2017-03-28 12:01 /local/snowplow/enriched-events/part-00073
-rw-r--r--   2 hadoop hadoop   45054205 2017-03-28 12:01 /local/snowplow/enriched-events/part-00074
-rw-r--r--   2 hadoop hadoop   44610558 2017-03-28 12:01 /local/snowplow/enriched-events/part-00075
-rw-r--r--   2 hadoop hadoop   44315616 2017-03-28 12:01 /local/snowplow/enriched-events/part-00076
-rw-r--r--   2 hadoop hadoop   44289826 2017-03-28 12:01 /local/snowplow/enriched-events/part-00077
-rw-r--r--   2 hadoop hadoop   44315367 2017-03-28 12:01 /local/snowplow/enriched-events/part-00078
-rw-r--r--   2 hadoop hadoop   44059023 2017-03-28 12:01 /local/snowplow/enriched-events/part-00079
-rw-r--r--   2 hadoop hadoop   43744203 2017-03-28 12:01 /local/snowplow/enriched-events/part-00080
-rw-r--r--   2 hadoop hadoop   43705582 2017-03-28 12:01 /local/snowplow/enriched-events/part-00081
-rw-r--r--   2 hadoop hadoop   43182651 2017-03-28 12:01 /local/snowplow/enriched-events/part-00082
-rw-r--r--   2 hadoop hadoop   42918074 2017-03-28 12:01 /local/snowplow/enriched-events/part-00083
-rw-r--r--   2 hadoop hadoop   42829920 2017-03-28 12:01 /local/snowplow/enriched-events/part-00084
-rw-r--r--   2 hadoop hadoop   42729196 2017-03-28 12:01 /local/snowplow/enriched-events/part-00085
-rw-r--r--   2 hadoop hadoop   42653259 2017-03-28 12:01 /local/snowplow/enriched-events/part-00086
-rw-r--r--   2 hadoop hadoop   42472337 2017-03-28 12:01 /local/snowplow/enriched-events/part-00087
-rw-r--r--   2 hadoop hadoop   42289965 2017-03-28 12:01 /local/snowplow/enriched-events/part-00088
-rw-r--r--   2 hadoop hadoop   41666320 2017-03-28 12:01 /local/snowplow/enriched-events/part-00089
-rw-r--r--   2 hadoop hadoop   41379765 2017-03-28 12:01 /local/snowplow/enriched-events/part-00090
-rw-r--r--   2 hadoop hadoop   41157277 2017-03-28 12:01 /local/snowplow/enriched-events/part-00091
-rw-r--r--   2 hadoop hadoop   40830954 2017-03-28 12:01 /local/snowplow/enriched-events/part-00092
-rw-r--r--   2 hadoop hadoop   40555186 2017-03-28 12:01 /local/snowplow/enriched-events/part-00093
-rw-r--r--   2 hadoop hadoop   40487401 2017-03-28 12:01 /local/snowplow/enriched-events/part-00094
-rw-r--r--   2 hadoop hadoop   40291258 2017-03-28 12:01 /local/snowplow/enriched-events/part-00095
-rw-r--r--   2 hadoop hadoop   40243445 2017-03-28 12:01 /local/snowplow/enriched-events/part-00096
-rw-r--r--   2 hadoop hadoop   39331423 2017-03-28 12:01 /local/snowplow/enriched-events/part-00097
-rw-r--r--   2 hadoop hadoop   38995214 2017-03-28 12:01 /local/snowplow/enriched-events/part-00098
-rw-r--r--   2 hadoop hadoop   38793520 2017-03-28 12:01 /local/snowplow/enriched-events/part-00099
-rw-r--r--   2 hadoop hadoop   38746118 2017-03-28 12:01 /local/snowplow/enriched-events/part-00100
-rw-r--r--   2 hadoop hadoop   38308873 2017-03-28 12:01 /local/snowplow/enriched-events/part-00101
-rw-r--r--   2 hadoop hadoop   38058958 2017-03-28 12:01 /local/snowplow/enriched-events/part-00102
-rw-r--r--   2 hadoop hadoop   37910410 2017-03-28 12:01 /local/snowplow/enriched-events/part-00103
-rw-r--r--   2 hadoop hadoop   37922536 2017-03-28 12:01 /local/snowplow/enriched-events/part-00104
-rw-r--r--   2 hadoop hadoop   37221601 2017-03-28 12:01 /local/snowplow/enriched-events/part-00105
-rw-r--r--   2 hadoop hadoop   37176983 2017-03-28 12:01 /local/snowplow/enriched-events/part-00106
-rw-r--r--   2 hadoop hadoop   36610008 2017-03-28 12:01 /local/snowplow/enriched-events/part-00107
-rw-r--r--   2 hadoop hadoop   36198816 2017-03-28 12:01 /local/snowplow/enriched-events/part-00108
-rw-r--r--   2 hadoop hadoop   36065512 2017-03-28 12:01 /local/snowplow/enriched-events/part-00109
-rw-r--r--   2 hadoop hadoop   36086007 2017-03-28 12:01 /local/snowplow/enriched-events/part-00110
-rw-r--r--   2 hadoop hadoop   36026644 2017-03-28 12:01 /local/snowplow/enriched-events/part-00111
-rw-r--r--   2 hadoop hadoop   35904984 2017-03-28 12:01 /local/snowplow/enriched-events/part-00112
-rw-r--r--   2 hadoop hadoop   35609774 2017-03-28 12:01 /local/snowplow/enriched-events/part-00113
-rw-r--r--   2 hadoop hadoop   35523393 2017-03-28 12:01 /local/snowplow/enriched-events/part-00114
-rw-r--r--   2 hadoop hadoop   34890156 2017-03-28 12:01 /local/snowplow/enriched-events/part-00115
-rw-r--r--   2 hadoop hadoop   34651030 2017-03-28 12:01 /local/snowplow/enriched-events/part-00116
-rw-r--r--   2 hadoop hadoop   34628877 2017-03-28 12:01 /local/snowplow/enriched-events/part-00117
-rw-r--r--   2 hadoop hadoop   34040834 2017-03-28 12:01 /local/snowplow/enriched-events/part-00118
-rw-r--r--   2 hadoop hadoop   32947126 2017-03-28 12:01 /local/snowplow/enriched-events/part-00119
-rw-r--r--   2 hadoop hadoop   32539900 2017-03-28 12:01 /local/snowplow/enriched-events/part-00120
-rw-r--r--   2 hadoop hadoop   32408952 2017-03-28 12:01 /local/snowplow/enriched-events/part-00121
-rw-r--r--   2 hadoop hadoop   32370653 2017-03-28 12:01 /local/snowplow/enriched-events/part-00122
-rw-r--r--   2 hadoop hadoop   32054664 2017-03-28 12:01 /local/snowplow/enriched-events/part-00123
-rw-r--r--   2 hadoop hadoop   32015904 2017-03-28 12:01 /local/snowplow/enriched-events/part-00124
-rw-r--r--   2 hadoop hadoop   31822903 2017-03-28 12:01 /local/snowplow/enriched-events/part-00125
-rw-r--r--   2 hadoop hadoop   31058732 2017-03-28 12:01 /local/snowplow/enriched-events/part-00126
-rw-r--r--   2 hadoop hadoop   30761032 2017-03-28 12:01 /local/snowplow/enriched-events/part-00127
-rw-r--r--   2 hadoop hadoop   30553632 2017-03-28 12:01 /local/snowplow/enriched-events/part-00128
-rw-r--r--   2 hadoop hadoop   30203546 2017-03-28 12:01 /local/snowplow/enriched-events/part-00129
-rw-r--r--   2 hadoop hadoop   29453906 2017-03-28 12:01 /local/snowplow/enriched-events/part-00130
-rw-r--r--   2 hadoop hadoop   29351679 2017-03-28 12:01 /local/snowplow/enriched-events/part-00131
-rw-r--r--   2 hadoop hadoop   28723025 2017-03-28 12:01 /local/snowplow/enriched-events/part-00132
-rw-r--r--   2 hadoop hadoop   28609890 2017-03-28 12:01 /local/snowplow/enriched-events/part-00133
-rw-r--r--   2 hadoop hadoop   28594433 2017-03-28 12:01 /local/snowplow/enriched-events/part-00134
-rw-r--r--   2 hadoop hadoop   27221636 2017-03-28 12:01 /local/snowplow/enriched-events/part-00135
-rw-r--r--   2 hadoop hadoop   26578153 2017-03-28 12:01 /local/snowplow/enriched-events/part-00136
-rw-r--r--   2 hadoop hadoop   26571390 2017-03-28 12:01 /local/snowplow/enriched-events/part-00137
-rw-r--r--   2 hadoop hadoop   26435524 2017-03-28 12:01 /local/snowplow/enriched-events/part-00138
-rw-r--r--   2 hadoop hadoop   25795347 2017-03-28 12:01 /local/snowplow/enriched-events/part-00139
-rw-r--r--   2 hadoop hadoop   24930606 2017-03-28 12:01 /local/snowplow/enriched-events/part-00140
-rw-r--r--   2 hadoop hadoop   21615009 2017-03-28 12:01 /local/snowplow/enriched-events/part-00141
-rw-r--r--   2 hadoop hadoop   20858684 2017-03-28 12:01 /local/snowplow/enriched-events/part-00142
-rw-r--r--   2 hadoop hadoop   17165094 2017-03-28 12:01 /local/snowplow/enriched-events/part-00143
-rw-r--r--   2 hadoop hadoop   17063982 2017-03-28 12:01 /local/snowplow/enriched-events/part-00144
-rw-r--r--   2 hadoop hadoop   16152820 2017-03-28 12:01 /local/snowplow/enriched-events/part-00145
-rw-r--r--   2 hadoop hadoop   15046982 2017-03-28 12:01 /local/snowplow/enriched-events/part-00146
-rw-r--r--   2 hadoop hadoop   11157380 2017-03-28 12:01 /local/snowplow/enriched-events/part-00147
-rw-r--r--   2 hadoop hadoop    7046866 2017-03-28 12:01 /local/snowplow/enriched-events/part-00148

#4

@alex seems to me there’s a bug either in iglu scala server or ilguctl code that pushes schemas to iglu scala server.

We recently added a few new event models to our stack. Some of them contained UTF-8 characters ( 山田 specifically )
The schemas were distributed to s3 repository and effectively picked up by scala hadoop enricher. Enricher completed, never faltered. On the other hand, scala hadoop shredder consistently failed until the schemas were republished without the character sequences above. Either igluctl does not include proper content-encoding headers, or iglu scalar server has a bug of some sort that causes downstream effects.

Do you have any suggestions on how to triangulate the issue? Any test suite I can use to validate iglu served documents?
Seems that corrupted shemas cause a runtime NPE.


#5

Hello @dashirov,

I created a ticket to explore this issue. Just to be sure: are these unicode symbols contained in JSON Schema content?

What happens if you try to fetch these schemas from scala server via curl? This seems especially strange because Enrich and Shred jobs use same Iglu Client and if one of them fails another has to fail consistently. What Enrich/Hadoop jobs versions are using?


#6

Versions:
hadoop_enrich: 1.8.0
hadoop_shred: 0.10.0 ( I’ve given 11RC1 a try but it bailed from the start, didn’t want to continue investigation with it)

  1. schema pushed to an s3 bucket with igluctl was not different from the original file
  2. schema pushed to iglu server and then retrieved via API and swagger UI was NOT identical when compared to the original.

Exact fields that I noticed a discrepancy on was:

    "first_name" : {
      "type": "string",
      "maxLength": 255,
      "example": "John",
      "description" : "Romanized given name (i.e. 山田 is not welcome, should be romanized to Yamada) stripped of surrounding white space"
},
"last_name":{
      "type": "string",
      "maxLength": 255,
      "example": "Smith",
      "description" : "Romanized surname / family name (i.e 太郎 is not welcome, should be romanized to Tarō) stripped of surrounding white space"
}

Discrepancies were against the following sequences: 山田, 太郎, Tarō
I did purge the potentially offending character sequences from the repository to restore the data processing pipeline. I’ll have to setup another environment to run tests against. Can you suggest a testing strategy ?


#7

@dashirov one way I see to reproduce error would be to run Scala Hadoop Shred tests with injected Iglu Registry (copy of Iglu Central that contains unicode characters), but I guess that is too much clutter if you’re not familiar with Scala and SHS code. I’ll try to reproduce it by myself tomorrow and will let you know about results.


#8

Hello @dashirov!

Unfortunately I did not discover anything suspicious while running Scala Hadoop Shred test suite with Schemas with unicode characters. Here’s what I did:

  • Inserted exact your description in several key Iglu schemas (particularly, iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0)
  • Uploaded these schemas along all others from Iglu Central to snowplow mini instance
  • Injected Iglu Server from Snowplow Mini as Iglu Central in test suite
  • Ran test suite

Tests passed as usually, without any failures. If I make any errors in contexts schemas - I immediately start to see bad rows, exceptions etc.

However, I agree that these symptoms may point us to unicode support bug, but at the same time I’m puzzled about why it doesn’t fail on enrich step and why it didn’t appear in my test. Also I’m almost sure I saw schemas with unicode symbols before (not on Iglu Scala Server though) and not aware of any problems.

Previously we encountered bugs in Iglu Scala Client where it silently throw an exception that didn’t appear in bad rows and didn’t short-circuit job, but I don’t think it can happen now with igluctl lint and Iglu Client improvements.


#9

Thank you very much for your time. I’ll dig a little more, if fruitful - will update.