Amazon SQS [ARCHIVED]
Overview
The Airbyte SQS destination allows you to sync data to Amazon SQS. It currently supports sending all streams to a single Queue.
Sync overview
Output schema
All streams will be output into a single SQS Queue.
Amazon SQS messages can only contain JSON, XML or text, and this connector supports writing messages in all three formats. See the Writing Text or XML messages section for more info.
Features
| Feature | Supported?(Yes/No) | Notes |
|---|---|---|
| Full Refresh Sync | No | |
| Incremental - Append Sync | Yes | |
| Incremental - Append + Deduped | No | |
| Namespaces | No |
Getting started
Requirements
- AWS IAM Access Key
- AWS IAM Secret Key
- AWS SQS Queue
Permissions
If the target SQS Queue is not public, you will need the following permissions on the Queue:
sqs:SendMessage
Properties
Required properties are 'Queue URL' and 'AWS Region' as noted in bold below.
- Queue URL (STRING)
- The full AWS endpoint URL of the queue
e.g.
https://sqs.eu-west-1.amazonaws.com/1234567890/example-queue-url
- The full AWS endpoint URL of the queue
e.g.
- AWS Region (STRING)
- The region code for the SQS Queue e.g. eu-west-1
- Message Delay (INT)
- Time in seconds that this message should be hidden from consumers.
- See the AWS SQS documentation for more detail.
- AWS IAM Access Key ID (STRING)
- The Access Key for the IAM User with permissions on this Queue
- Permission
sqs:SendMessageis required
- AWS IAM Secret Key (STRING)
- The Secret Key for the IAM User with permissions on this Queue
- Message Body Key (STRING)
- Rather than sending the entire Record as the Message Body, use this property to reference a Key in the Record to use as the message body. The value of this property should be the Key name in the input Record. The key must be at the top level of the Record, nested Keys are not supported.
- Message Group Id (STRING)
- When using a FIFO queue, this property is required.
- See the AWS SQS documentation for more detail.
Setup guide
Using the Message Body Key
This property allows you to reference a Key within the input Record as using that properties Value as the SQS Message Body.
For example, with the input Record:
{
"parent_with_child": {
"child": "child_value"
},
"parent": "parent_value"
}
To send only the parent_with_child object, we can set Message Body Key to parent_with_child.
Giving an output SQS Message of:
{
"child": "child_value"
}
Writing Text or XML messages
To output Text or XML, the data must be contained within a String field in the input data, and then
referenced by setting the Message Body Key property.
For example, with an input Record as:
{
"my_xml_field": "<something>value</something>"
}
To send a pure XML message, you would set the Message Body Key to my_xml_field.
The output SQS message would contain:
<something>value</something>