Enricher Buffer Going above 500 causing Kinesis ValidationException

For engineers Enrichment
1

HI,
Situation: Running a snowplow_stream_enrich_kinesis_0.15.0 with buffer conf as

      byteLimit = 20000000
      recordLimit = 480 
      timeLimit = 60000

I am getting

 'records' failed to satisfy constraint: Member must have length less than or equal to 500 (Service: AmazonKinesis; Status Code: 400; Error Code: ValidationException;

Which I can see is because it is trying to write more than 500 in my case 502 records to Kinesis.

But since the buffer setting prohibits it from going above 480, this shouldn’t happen, right?

I have changed it to a much lover size and time and not seeing this error anymore, but it shouldn’t have happened right?

Could you explain whats happening here? is it approx buffer sizes?

2

At the moment a flush is triggered when one of those (whichever occurs first) limits is it. At the moment there’s nothing to stop more than 480 (or 500) from being sent through and this could occur in some scenarios including retrying failed PutRecords attempts.

3

yep i think i could see some retries before this 500 started occuring .
Thanks for the clarification