Kinesis Stream Transformer not writing to AWS SQS

Hey Snowplow, we are currently following your Kinesis Stream Transformer docs and we are encountering some issues with the SQS queue.

Problem

We are able to send test events through the pipeline (collector → enricher → transformer) and see these written to S3, however, we are not observing any messages in the configured SQS queue.

Tried & Failed Fixes

  • Given the ECS task running the transformer very open permissions.
  • Attached a very open resource-based-policy to the SQS queue.

Errors Encountered

software.amazon.awssdk.services.sqs.model.SqsException: Value shredding for parameter MessageGroupId is invalid. Reason: The request include parameter that is not valid for this queue type. (Service: Sqs, Status Code: 400, Request ID: 95402e98-c015-5186-bf35-75f9014fa56c, Extended Request ID: null)

Relevant Configuration

{ "input": { "type": "kinesis", "appName": ${?DDB_NAME}, "streamName": ${?INPUT_STREAM_NAME}, "region": "eu-west-1", "position": "TRIM_HORIZON" "retrievalMode": { "type": "Polling" "maxRecords": 10000 } "bufferSize": 3 } "output": { "path": ${?TRANSFORMED_ARCHIVE_PATH}, "compression": "GZIP", "region": "eu-west-1" } "windowing": "5 minutes" "queue": { "type": "sqs", "queueName": ${?SQS_QUEUE_NAME}, "region": "eu-west-1" } "formats": { "transformationType": "widerow", "default": "TSV", } "monitoring": { "metrics": { "cloudwatch": true } } … }

With DDB_NAME (name of DynamoDB table - not ARN), INPUT_STREAM_NAME (name of the input Kinesis Stream - not ARN), TRANSFORMED_ARCHIVE_PATH (S3 URI) and SQS_QUEUE_NAME (name of the SQS queue - not ARN) configured as environment variables.Are you able to help us diagnose this issue? Let us know if you require any additional information. Thanks

Hi @Daniel_Baron,

Can you confirm your SQS is FIFO ? If it is not, can you try to use FIFO one, please ?

Seems like we’ve missed to specify it in our docs. We will fix it.

1 Like

Hey @enes_aldemir

This was exactly it! Thanks for getting back to us os quickly!

Seems like we’ve missed to specify it in our docs. We will fix it.

This would be really good thanks!

2 Likes