feat(new sink): add AWS Simple Notification Service

[wochinge]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

• We want to use vector together with AWS Simple Notification Service to write events to multiple destinations within AWS at the same time. In our case we have multiple consumers of the data:
  • simple persistence in database
  • multiple consumers processing the events and running calculations on top of it
• We want to reduce coupling between the vector sink and the topic subscribers
• One could probably also argue that
  • SNS is the better SQS sink since it's more flexible (it's quite easy to subscribe a SQS queue to a SNS topic).
  • Kafka alternative for the AWS ecosystem (would very nicely complement the other available sinks)

Attempted Solutions

• Use AWS EventBridge pipes + AWS SQS to achieve the same: Would be quite nice but we want to send the message to different topics (we're using vector in the sidecar pattern). To achieve this with the EventBridge we would require a lot of queues and lot of AWS EventBridge pipes.
• Use the vector HTTP sink and send it to AWS SNS API: could in theory also work but would require a lot of patching of the event to make the request payload compliant with the API and then we'd still need to handle the AWS authentication...

Proposal

I've already added a draft PR with a suggested implementation. Since it's very similar to the AWS sink we can re-use almost all of the things from the SQS implementation (testing environment, a lot of code, documentation, etc.). In line with that I'd also keep documentation / configuration very similar to AWS SQS.

References

• feat(new sink): add AWS Simple Notification Service aws_sns sink #18141

Version

v0.31.0

[neuronull]

Hi @wochinge, thanks for suggesting the sink and adding details about the use case and alternative solutions.

One question so far is- would you personally (or if you are proposing this new component for the benefit of a business, would that entity) be open to an informal agreement to support the sink going forward? i.e , if there are bugs or enhancements filed about this new component, being open to being tagged as the assignee/reviewer for those issues.

Thanks!

[wochinge]

@neuronull

We would like to use this new sink as part of an essential production component in my company. As part of that we of course have an interest in having this component in vector as well as fixing any bugs within the component or doing improvements. Happy to help out the community and contribute back (+ it gives me a reason to learn Rust :-)). I can relatively confidently say that this holds true for the next year but everything beyond that would be just guessing.

From the AWS ecosystem perspective I also think that maintaining the SNS sink makes in my opinion more sense than maintaining the SQS sink as that's the more versatile approach.

[neuronull]

Thanks for sharing that background info, and for being open to supporting it.

There may be further follow up questions but for now we will take a little time to work through some internal processing of the new component request. Stay tuned ~

[neuronull]

Good news- we're open to accepting this new component 🎉

Given the amount of overlap with the aws_sqs sink, we'll be looking to re-use as much code as possible. I'll post a comment in the PR about this.

[wochinge]

Great news! 🎉 Thanks, @neuronull!