Connection reset by peer on load testing on scala stream collector


#1

Hi Snowplow,
i was trying to load testing my collector using avalanche, here is some details about it :

  • Ubuntu Xenial 16 ( Desktop Edition )
  • Java 1.7 - Scala stream collector 0.7.0
  • Avalanche with exponentialpeak with env
    SP_SIM_TIME=10
    SP_BASELINE_USERS=500
    SP_PEAK_USERS=1500

Errors That came up:
15:26:01.491 [WARN ] i.g.h.a.ResponseProcessor - Request ‘PageView Event’ failed: j.i.IOException: Connection reset by peer

i cannot figure out why this is happening,and the errors seems occurs randomly, sometimes they give me some errors and sometimes there’s no error at all. I notice at the time the error came, my cpu usage isn’t high around 60% to 80% of CPU usage and around 80% of RAM usage, can you help me figure out what cause this problem?

Thank You,


#2

Hi @ChocoPowwwa,

On the server running the Collector what is your current open files limit? If the server still had headroom it shouldn’t be refusing connections but this sounds like some hard-limit on the server has blocked the connection coming in.

The other thing to consider is that the collector needs to push all of this information of to Kinesis, if for whatever reason that process slows down temporarily or if the collector cannot push fast enough to Kinesis due to a shard limitation it might not be able to accept incoming connections until that pressure is eased.

Ways around this are too:

  1. Add more nodes to your collector cluster to distribute load
  2. Ensure there is sufficient headroom in your stream to handle the load
  3. Ensure your server has a high open-files limit so that it can handle more open connections (incoming and out-going)

Your avalanche simulation is highly very short, you are going from no load up to 2000 users which is in essence 2000 requests/second - it is possible that even though the hardware does not look maxed out it simply cannot push the information out quickly enough.

Hope this helps,
Josh


#3

Hi @josh , thanks for answering the issue,

Sorry my bad, you’re right, i wasn’t have enough headroom on my machine, i did increase the CPU and ram and it works fine, sorry for my dumb question, thank you.