I have setup snowplow real time.I want to test some sample events .
I want to check whether events are received by collector and able to get the same in kibana dashboard.Is there any way to trigger events to snowplow collector without using tracker.
I’m using below commands to check:
for collector check:htttp://elbendpoint/health
for sample empty event :http://endpoint/i?
but im not sure about events coming to bad index or good index after hitting http://endpoint/i?
these are errors im getting in elastic loader configured for bad index load in kibana
[RecordProcessor-0000] ERROR com.snowplowanalytics.elasticsearch.loader.clients.ElasticsearchSenderHTTP - Shutting down application as unable to connect to Elasticsearch for over 10000000 ms
org.elasticsearch.client.ResponseException: POST http://search.test_elasticsearch.us-east-1.es.amazonaws.com:443/_bulk: HTTP/1.1 400 Bad Request
^M
400 The plain HTTP request was sent to HTTPS port^M
^M
400 Bad Request
^M
The plain HTTP request was sent to HTTPS port^M
^M
^M
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:354)
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:343)
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
at java.base/java.lang.Thread.run(Thread.java:834)
[main] INFO com.sksamuel.elastic4s.http.HttpClient$ - Creating HTTP client on http://search-bbsnowplow.us-east-1.es.amazonaws.com:443
[main] INFO com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator - With failover time 30000 ms and epsilon 25 ms, LeaseCoordinator will renew leases every 9975 ms, takeleases every 60050 ms, process maximum of 2147483647 leases and steal 1 lease(s) at a time.
[main] WARN com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Received configuration for both region name as us-east-1, and Amazon Kinesis endpoint as https://kinesis.us-east-1.amazonaws.com. Amazon Kinesis endpoint will overwrite region name.
[main] INFO com.snowplowanalytics.elasticsearch.loader.KinesisSourceExecutor - KinesisSourceExecutor worker created
[main] INFO com.amazonaws.services.kinesis.connectors.KinesisConnectorExecutorBase - Starting worker in KinesisSourceExecutor
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initialization attempt 1
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initializing LeaseCoordinator
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Syncing Kinesis shard info
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Starting LeaseCoordinator
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Initialization complete. Starting worker loop.
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - No activities assigned
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Sleeping …
[LeaseCoordinator-2] INFO com.amazonaws.services.kinesis.leases.impl.LeaseTaker - Worker 10393a688486a28f:55e3611d:16eb5bb2528:-8000 saw 1 total leases, 1 available leases, 1 workers. Target is 1 leases, I have 0 leases, I will take 1 leases
[LeaseCoordinator-2] INFO com.amazonaws.services.kinesis.leases.impl.LeaseTaker - Worker 10393a688486a28f:55e3611d:16eb5bb2528:-8000 successfully took 1 leases: shardId-000000000000
[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Created new shardConsumer for : ShardInfo [shardId=shardId-000000000000, concurrencyToken=8bc40f93-37d4-431c-9940-489d5f585399, parentShardIds=, checkpoint={SequenceNumber: TRIM_HORIZON,SubsequenceNumber: 0}]
[RecordProcessor-0000] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.BlockOnParentShardTask - No need to block on parents of shard shardId-000000000000
[RecordProcessor-0000] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.KinesisDataFetcher - Initializing shard shardId-000000000000 with TRIM_HORIZON
[pool-2-thread-1] ERROR com.snowplowanalytics.elasticsearch.loader.clients.ElasticsearchSenderHTTP - ElasticsearchSender threw an unexpected exception
org.elasticsearch.client.ResponseException: POST http://search-bbsnowplow.us-east-1.es.amazonaws.com:443/_bulk: HTTP/1.1 400 Bad Request
^M
400 The plain HTTP request was sent to HTTPS port^M
^M
400 Bad Request
^M
The plain HTTP request was sent to HTTPS port^M
^M
^M
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:354)
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:343)
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
Looks to me like there’s something wrong in connecting with Elasticsearch - it’s not an area I’m particularly strong on, but my guess from the logs it that it could be a networking issue.
