Situation: Running a
snowplow_stream_enrich_kinesis_0.15.0 with buffer conf as
byteLimit = 20000000 recordLimit = 480 timeLimit = 60000
I am getting
'records' failed to satisfy constraint: Member must have length less than or equal to 500 (Service: AmazonKinesis; Status Code: 400; Error Code: ValidationException;
Which I can see is because it is trying to write more than 500 in my case 502 records to Kinesis.
But since the buffer setting prohibits it from going above 480, this shouldn’t happen, right?
I have changed it to a much lover size and time and not seeing this error anymore, but it shouldn’t have happened right?
Could you explain whats happening here? is it approx buffer sizes?