Payload Size too big - ruby


#1

we have a pretty big payload coming from Ruby tracker. and getting 414 errors on collector of course.

Javascript mentions that it’s able to split it up on it’s own if it’s too big. do we have this option in Ruby as well? we have one big event that we can split into multiple events to keep the payload size down, but quite a bit of work to do so, so wanted to check with you first.


#2

How large is your payload (bytes / characters) at the moment?

Are you using the Scala Stream collector or the Clojure collector?


#3

@mike stream collector
one sample looks like 13271 characters

java -jar snowplow-stream-collector-0.9.0 --config collector.conf


#4

The max URI in the configuration for stream collector is 32768 characters so it shouldn’t be returning a 414 for less than this. Can you try with the latest version of stream collector?


#5

@mike yes thanks!


#6

@mike weird though, looking at current collector.conf, it has this

parsing {
max-uri-length = 32768
}


#7

@mike upgraded to latest collector on dev and still error code 414.

i think this is a problem on the nginx side of things. i’m going to bump the URI length there and restart and test again.

8K chars:
1 1 8190 /tmp/tmp


#8

tried upping config on nginx and now i get the right payload size, but still code 414

77004 HTTP/1.1" 414 58 “-” “Ruby”

1 13 17253 /tmp/tmp3

so 17K chars now. which matches the payload size ruby is sending from server.


#9

@mike we figured it out.

When we change Ruby client to do a POST instead of GET, it’s fine.


#10

Great!

How are you running nginx out of interest? Are you using this as a load balancer in front of the scala application itself?


#11

@mike yup