Cloudwatch Throttling


#1

Hey Snowplow team, We recently hit a problem with our S3 sync - getting the error of “Could not publish N datums to CloudWatch”

I am still reviewing the code of the S3 sync trying to figure out what the publishMetrics to CloudWatch has to do with syncing to S3.

Here’s the error:

Jul 29, 2016 2:55:54 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
WARNING: Could not publish 8 datums to CloudWatch
com.amazonaws.AmazonServiceException: Rate exceeded (Service: AmazonCloudWatch; Status Code: 400; Error Code: Throttling; Request ID: {})
	at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1160)
	at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:748)
	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302)
	at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:982)
	at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.putMetricData(AmazonCloudWatchClient.java:370)
	at com.amazonaws.services.kinesis.metrics.impl.DefaultCWMetricsPublisher.publishMetrics(DefaultCWMetricsPublisher.java:63)
	at com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable.runOnce(CWPublisherRunnable.java:144)
	at com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable.run(CWPublisherRunnable.java:90)
	at java.lang.Thread.run(Thread.java:745)

#2

Also, any luck on removing this or making it abide by the logging rate in the config file?

com.snowplowanalytics.snowplow.storage.kinesis.s3.RawEventTransformer toClass
INFO: Converting one record to EmitterInput before adding it to the buffer

https://github.com/snowplow/kinesis-s3/issues/50


#3

Hey @13scoobie:

  • We’ve seen that CloudWatch warning before - but it’s just a warning as I understand it, it doesn’t impact rotation to S3
  • Yes, that ticket regarding reducing INFO logging should go into the next Kinesis S3 release

#4

sounds good! Looking forward to it. We had an issue where the S3 sync was crashing and were trying to debug, but trying to sift through the thousands of lines with RawEventTransformer made it difficult. Glad to hear that a fix is coming!