Stream Collector process dead randomly


#1

Hi,

My stream collector process dead randomly. I can see process run in background and port be listening, but it lost response randomly. I need restarrt it to make it works again.

Collector still run at background when it crashed

root 1230992 2.3 4.2 5151448 254080 pts/0 Sl 00:43 0:33 java -jar snowplow-stream-collector-0.9.0 --config collector.conf
root 1230993 0.7 2.4 4404476 146976 pts/0 Sl 00:43 0:10 java -jar snowplow-kinesis-s3-0.4.0 --config kinesis-s3.conf

Collector still listen at right when it crashed

tcp 0 0 123.123.123.123:80 :::* LISTEN 1230992/java

Error message at http log:

[proxy_http:error] …Connection reset by peer: … AH01102: error reading status line from remote server 123.123.123.123:80, referer: https://abc.abc.com/

I try check http active from localhost. When collector works,checking message like this:
HTTP WARNING: HTTP/1.1 400 Bad Request - 185 bytes in 0.057 second response time |time=0.056884s;;;0.000000 size=185B;;;0

When collector stop working,checking message like this:
HTTP CRITICAL - No data received from host

Can anyone help? Thanks


#2

when crash happened,collector script out put like this:

INFO: Sleeping …
Feb 16, 2017 1:28:52 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:29:02 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:29:13 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:29:23 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 14 datums.
Feb 16, 2017 1:29:33 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:29:43 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:29:52 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker info
INFO: No activities assigned
Feb 16, 2017 1:29:52 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker info
INFO: Sleeping …
Feb 16, 2017 1:29:54 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:30:04 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:30:14 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 14 datums.
Feb 16, 2017 1:30:24 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:30:34 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:30:45 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:30:53 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker info
INFO: No activities assigned
Feb 16, 2017 1:30:53 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker info
INFO: Sleeping …
Feb 16, 2017 1:30:55 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:31:05 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:31:15 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 14 datums.
Feb 16, 2017 1:31:26 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:31:36 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:31:46 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:31:54 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker info
INFO: No activities assigned
Feb 16, 2017 1:31:54 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker info
INFO: Sleeping …
Feb 16, 2017 1:31:56 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:32:07 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:32:17 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 14 datums.
Feb 16, 2017 1:32:27 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:32:37 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.
Feb 16, 2017 1:32:47 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 4 datums.

AWS Collector Kinesis Stream monitor show no active during crash time


#3

Hi @cyclx1,

There is a number of possible issues. Most popular ones are not sufficient ram memory, consumed all the cpu credits (we noticed it on EC2 and container service on AWS). Another possibility is a proxy/load balancer or another intermediate breaking your requests.


#4

Thanks for reply. I try to use 6G memory centos 7 server and there are lots of free space.

free -m
total used free shared buff/cache available
Mem: 5807 1962 890 291 2954 3283
Swap: 5119 0 5119

I dont know its proxy or not. but the collector stop response at localhost when crash happen