Error while uploading snowplow logs to s3

For engineers Data store sources
1

I am running snowplow-kinesis-s3-0.4.0 to upload logs to s3 but i am getting an error,

“com.amazonaws.AmazonClientException: Unable to execute HTTP request: s3-http”

not sure, what is the reason of the error because l am following your setup guide step by step
Config

sink {
aws {
access-key: “key”
secret-key: “key”
}

kinesis {
in {
stream-name: “good”
initial-position: “TRIM_HORIZON”
max-records: “1000”
}

out {
  stream-name: "bad"
}

region: "us-west-2"

app-name: "snownrich"

}

s3 {
region: “http://s3-us-west-2.amazonaws.com/”
bucket: “bucketname/folder”
max-timeout: “300”
format: “lzo”
}

buffer {
byte-limit: 4500000
record-limit: 500 # Not supported by Kafka; will be ignored
time-limit: 60000
}
logging {
level: “error”
}

monitoring {
snowplow {
collector-uri: “xx.xx.xx.xx”
collector-port: 80
app-id: “collector-monitor”
method: “GET”
}
}
}

Error

com.snowplowanalytics.snowplow.storage.kinesis.s3.S3Emitter com$snowplowanalytics$snowplow$storage$kinesis$s3$S3Emitter$$attemptEmit$1
SEVERE: S3Emitter threw an unexpected exception
com.amazonaws.AmazonClientException: Unable to execute HTTP request: s3-http
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:478)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3672)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1436)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1302)
at com.snowplowanalytics.snowplow.storage.kinesis.s3.S3Emitter.com$snowplowanalytics$snowplow$storage$kinesis$s3$S3Emitter$$attemptEmit$1(S3Emitter.scala:157)
at com.snowplowanalytics.snowplow.storage.kinesis.s3.S3Emitter$$anonfun$emit$1.apply(S3Emitter.scala:189)
at com.snowplowanalytics.snowplow.storage.kinesis.s3.S3Emitter$$anonfun$emit$1.apply(S3Emitter.scala:189)
at scala.collection.immutable.List.foreach(List.scala:318)
at com.snowplowanalytics.snowplow.storage.kinesis.s3.S3Emitter.emit(S3Emitter.scala:189)
at com.amazonaws.services.kinesis.connectors.KinesisConnectorRecordProcessor.emit(KinesisConnectorRecordProcessor.java:159)
at com.amazonaws.services.kinesis.connectors.KinesisConnectorRecordProcessor.processRecords(KinesisConnectorRecordProcessor.java:132)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.V1ToV2RecordProcessorAdapter.processRecords(V1ToV2RecordProcessorAdapter.java:42)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.call(ProcessTask.java:172)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:48)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:23)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: s3-http
at java.net.InetAddress.getAllByName0(InetAddress.java:1273)
at java.net.InetAddress.getAllByName(InetAddress.java:1185)
at java.net.InetAddress.getAllByName(InetAddress.java:1119)
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:44)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:259)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:159)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:706)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467)
… 19 more

Feb 22, 2017 7:29:24 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 13 datums.
Feb 22, 2017 7:29:28 AM com.snowplowanalytics.snowplow.storage.kinesis.s3.S3Emitter forceShutdown
SEVERE: Shutting down application as unable to connect to S3 for over 300 ms

2

Does s3 bucket exist in the specified region? Does your user have privilages to write to the bucket? This looks like privilages issue…

3

no its not privileges issue because i just have uploaded files to my s3 bucket using same IAM keys using s3cmd.
region is also correct: us-west-2

4

Can you verify form the instance running kinesis-s3 storage to verify if s3 endpoint is not firewalled?

5

hey, i just have tested the s3cmd on the same server where i am running

command:
./s3cmd ls s3://snowplow-xxx-xxx/

output:

DIR s3://snowplow-xxx-xxx/logs/
2017-02-22 07:26 16606 s3://snowplow-xxx-xxx/logo.png

s3cmd config:
region: us-west-2
using same IAM id and key

if issue was because of firewall, i wont be able to ls or upload file’s to s3 from same server using s3cmd

6

i am not sure about this but following error is

Feb 22, 2017 6:17:02 PM com.amazonaws.http.AmazonHttpClient executeHelper
INFO: Unable to execute HTTP request: snowplowlogbucket.s3-http

java.net.UnknownHostException: snowplowlogbucket.s3-http

but it should be some complete aws s3 URL instead of just snowplowlogbucket.s3-http.

maybe that’s why snowplow-kinesis-s3-0.4.0 si not able to execute http request.

7

Hi @geetanshjindal - from your config you have put region: "http://s3-us-west-2.amazonaws.com/" - this should simply be “us-west-2”.

Could you please update that and try again?

8

yes its working now, Thanks.