Building a Serverless Newsletter: Your Guide to AWS and Amazon SES
Discover the secrets of creating a serverless newsletter using AWS and Amazon SES. Dive into SST, Middy with Typescript, and master everything you need to know about SES for a seamless newsletter service.
ā ļø Hey! This blog post was initially published on my own blog. Check it out from the source: https://cloudnature.net/blog/building-a-serverless-newsletter-your-guide-to-aws-and-amazon-ses
Introduction
Have you ever wrestled with Amazon SES, firing off emails left and right, dealing with bounces, and handling complaints? I sure have. Then, a lightbulb moment hit š”, I want to build my own newsletter infrastructure! Sure, there are quicker routes out there, but whereās the fun in that?
So, buckle up for this blog post. Weāre delving into serverless setups and everything there is to know about Amazon SES. Of course, weāre making things work at scale!
And hold tight, because one section is titled āLambda? Nope, thank you!ā. Can serverless exists without Lambda functions? š¤
Project Overview
Creating a newsletter infrastructure is quite a puzzle. It needs to be quick, handle heavy traffic, and stay operational no matter what. So, why did I opt for a complete serverless approach? Itās straightforward: serverless computing is the superhero of performance, resilience, and scalability š¦ø.
And hey, itās also budget-friendly. Newsletters donāt need to be active 24/7; they wake up once a day, get the job done, and then rest until the next day. Letās break down the reasons behind choosing a serverless setup for this newsletter. But before we get there, letās tackle the elephant in the room by exploring three ways to create a newsletter.
The Fastest Way
Do I really have to spell it out? Numerous newsletter providers exist, so many choices, really. š
The Simple Way
Setting up a basic serverless newsletter infrastructure on AWS is a breeze, requiring just a few essential components:
- Validate your sender identity in Amazon SES
- Implement a straightforward AWS Step Function for sending emails with AWS
- Set up a single web service to publish the newsletter
With these in place, you can kick off your newsletter distribution. The beauty of this solution? In just 8ā16 hours, youāve built your custom newsletter infrastructure.
But hold on, weāre not aiming for a quick fix here, are we? Weāre on a mission to dedicate an entire weekend just to the HTML and CSS š. Donāt worry, Iām not trying to spook anyone. I believe in your champion-level skills, capable of centering a div better than I ever could.
Donāt worry, Iāll spare you the HTML and CSS details of the newsletter templates.
Letās shift our focus to the complete AWS infrastructure, which is the reason weāre all gathered here š.
The Complete Way
Building a serverless infrastructure doesnāt have to be a complete mess. Iām not a fan of unnecessary complexity. Instead, letās keep it simple by assembling basic components, creating completeness, and minimizing confusion.
So, what are these foundational bricks? Letās break them down:
- Setting up Amazon SES
- Additional Amazon SES configurations for Bounces, Complaints, Clicks (donāt worry, weāll dive into this shortly)
- API Web Services (AWS š) for seamless newsletter operations
- AWS Step function workflow for sending emails
These bricks, when glued together, form the backbone of our newsletter.
Now, letās delve into each element while keeping things clear and straightforward.
Amazon SES Setup
Setting up SES (Simple Email Service) is the starting point for our newsletter infrastructure. Itās the AWS service that powers our email delivery. Letās look at the infrastructure needed to send emails with AWS.
Domain Validation
Before we jump into the technicalities, letās get our domain validated. Why? Email providers want to know who owns that email address. The drill? Domain validation through DNS (basically, adding a few CNAME records in your DNS provider). In my case, a quick validation of https://cloudnature.net in Route53, and weāre good to go.
Find the full identity validation process in the Infrastructure As Code section at the bottom of the GitHub repository.
But hold on, validation alone is not enough for email providers. To be top-tier among other email providers, we need a bit more from Amazon SES:
- DKIM (DomainKeys Identified Mail)
- Custom MAIL FROM (cue SPF authentication and DMARC compliance)
- BIMI compliance
Now, letās look into all of them. The names might be a mouthful, but theyāre harmless, I promise š
DomainKeys Identified Mail (DKIM)
Ensures email authenticity by adding a digital signature. This is your emailās passport to a trustworthy inbox. DKIM is a security standard, ensures that an email claiming to be from a specific domain is genuinely authorized by the domain owner.
Opting for Easy DKIM means Amazon SES takes the lead. It generates a public-private key pair and includes a DKIM signature in every message sent from that identity. Your job? Well, nothing much, Amazon SES has got it covered!
Custom MAIL FROM
Ever wondered about the addresses behind the emails you send? Thereās the From address, visible to the recipient, and then thereās the MAIL FROM address, indicating the messageās origin. By default, Amazon SES assigns a MAIL FROM domain for your outgoing messages, using a subdomain of amazonses.com.
But what if you want to add a personal touch to your MAIL FROM address? As seen in the picture above, adding your own MAIL FROM means you get rid of that ugly āvia amazonses.comā tag near your name!
For this, you simply need to comply with the Sender Policy Framework (SPF), an email validation standard designed to prevent email spoofing. While at it, letās add another protocol: Domain-based Message Authentication, Reporting, and Conformance (DMARC). DMARC, fancy name, simple job: it helps detect email spoofing, allowing you to guide the email provider on how to act.
To check your DMARC compliance, visit š DMARC Inspector.
Brand Indicators for Message Identification (BIMI)
Give your emails a visual identity. Letās set up BIMI and become a recognized face in your recipientās inbox.
At this point, weāre almost there. But wouldnāt it be nice to have a logo as your email profile image? Thatās where BIMI comes in. If youāve followed the previous steps, your Amazon SES configuration is nearly ready.
All we need are two things:
- Create an SVG for our profile image.
- Add the BIMI record to our DNS registrar.
To convert your SVG logo (modern SVGs may not be sufficient for BIMI), download a simple yet efficient program in GitHub repository https://github.com/authindicators/svg-ps-converters/tree/master.
Afterward, upload your logo and add a DNS record like so:
default._bimi.dev.cloudnature.net TXT v=BIMI1;l=https://cloudnature.net/logos/logo.svg;
And there you have it! Check your BIMI compliance at š Bimi Group.
Email Sending Challenges
Sending emails may seem like a straightforward task, but the journey from your server to your recipientās inbox can be a bumpy ride. In this section, weāll navigate through common challenges encountered while sending emails.
We need to be careful because AWS policies are strict, so manual checking may not be enough. Hereās how our automatism flows:
- Some triggering event arrives
- Amazon SES triggers Amazon SNS
- Amazon SNS triggers AWS Lambda
- AWS Lambda run custom logic to decide if the email address needs to be removed from the newsletter list (for example if the email is unexistent), if so, it remove the record from the DynamoDB table
Letās see briefly one case at the time š.
Handling Bounces
Bounces in the email world are annoying but more common than you could think. They happen when an email cannot be delivered to the recipient, either due to an invalid email address or a temporary issue. In our serverless setup, dealing with bounces is crucial for maintaining a clean and effective newsletter infrastructure.
So, whatās the drill when a bounce event pops up? There are different types of bounces, but for the hard ones, we remove that email address from our newsletter list.
Addressing Complaints
Not every email recipient is thrilled to receive newsletters, and complaints may arise. Complaints occur when a recipient marks your email as spam or unwanted. Managing complaints is key to maintaining a positive sender reputation and ensuring your emails land in the inbox.
Now, what happen when a complaint event lands in our logs? Developers donāt pay attention to warnings, so treat it like a yellow card and investigate why it got the spam label. If it becomes a recurring event, consider removing that email address from your subscribers list.
To check if you are following every best practices, use this website: Mail Tester
Troubleshooting Errors
In the intricate world of email sending, errors can pop up unexpectedly. From server glitches to misconfigurations, troubleshooting errors is an essential part of maintaining a reliable newsletter infrastructure.
Always keep a watchful eye on error logging. Itās like your troubleshooting toolkit, it helps you identify issues worth fixing.
Tracking Clicks Per Link
Knowing how recipients engage with your newsletters is gold. Tracking clicks per links gives you the insight on the effectiveness of your email campaigns. While Amazon SES doesnāt offer this out of the box, weāve got to add to our serverless architecture an additional SNS Topic and an AWS Lambda. This duo saves records in our Amazon DynamoDB database, tracking every piece of information about a specific link.
A big shoutout to Guillermo Ojeda for enlightening me on the significance of this metric during our conversation about must-have metrics for newsletters. Thanks Guillermo š.
Newsletter API Web Services
Building the AWS serverless infrastructure was a piece of cache (sorry, I couldnāt resist). Letās break down the stack:
- SST with CDK integration
- Typescript
- Middy
These tools are the unsung heroes that took my developer experience to a whole new level. I wonāt dive too deep here; you can find the full repository with all the goodies at the bottom.
Now, which APIs are we talking about?
- CRUD APIs for admin operations on newsletter items
- Newsletter publish and unpublish APIs
- Newsletter subscribe and unsubscribe APIs
Pretty standard stuff, right? For the Amazon DynamoDB connection, I opted for dynamodb-toolbox: a library that proved its worth and is likely to feature in my future projects too.
Take a peek at a snippet of the Postman APIs developed for this project. One API in the mix serves as the documentation. If youāre authenticated and give it a buzz, itāll hand you the JSON Postman collection, all decked out with documentation ready to use (just remember to include environment variables).
AWS Lambda functions? No, Thank You!
Alright, letās kick off this chapter by addressing the elephant in the serverless room: AWS Lambda functions. Now, you might wonder, āWhy didnāt we use Lambda functions?ā Well, the answer is as simple as our serverless setup: a single AWS Step Function took the reins and orchestrated the entire show. Say goodbye to the need for countless Lambda functions or a single AWS Lambda function doing all the dirty work. Interested? Letās explore why one not-so-big AWS Step function stole the spotlight.
This overview divides our step function into three main parts:
- Workflow that sends the first email: a critical step to ensure everything is functioning as expected. An initial test to our own email address helps us validate the integrity of our setup.
- List every subscriber from Amazon DynamoDB: an essential task where we fetch the list of subscribers and organize them into batches of 50 email addresses each.
- For each batch, send out emails with Amazon SES: the final leg of our journey involves the systematic sending of emails in batches using Amazon SES.
Now, letās dive into each step and explore the challenges that may arise when building this intricate workflow with AWS Step Functions.
Sending Test Email
This segment involves a straightforward process. We retrieve the newsletter item from Amazon DynamoDB, process and convert it into a valid JSON, and then proceed to send the email using the Amazon SES SendBulkEmail V2 API.
SendBulkEmail stands out as the best practice, ensuring that each recipient sees only their own name and email address in the To header of the messages they receive.
This step proves to be exceptionally useful to know how the newsletter will truly appear in our email inbox. The simplicity and effectiveness of this approach allow for quick identification and correction if any HTML issues arise. Following this, we can unpublish and reschedule the newsletter publish date.
Processing Subscribers
This phase proved to be the most intricate to develop! Pagination for Amazon DynamoDB works differently compared to many NoSQL or SQL databases. You simply inform the database about the last item you retrieved. Using that as a starting point, we can list through every single subscriber.
The objective here is to produce a list of subscribed email addresses. However, to create that list, we need to go through a few rounds of Pass states. In my case, this was the workflow:
- First Step: Query elements from Amazon DynamoDB (not scan)
- Second Step: Create an array of items that merges previous items and new ones: States.Array($.dynamodbConfig.items[ * ], $.subscribers.Items[ * ].email.S)
- Third Step: The result is pretty ugly:
"items": [
[
"emailaddress3@example.com"
],
[
"emailaddress@example.com",
"emailaddress1@example.com",
"emailaddress2@example.com"
]
]
This means we need to flatten this list like so: $.dynamodbConfig.items[ * ][ * ]
- Fourth Step: Check if there are more elements. If not, we are going to close this process. If, in fact, there are more subscribers, we save the items in another object and start all over again.
Sending Out Emails
Now, we are almost ready to āhitā that send button and dispatch our newsletters. The workflow here is comparatively simpler than the previous one, but we need to make some adjustments to ensure we can call the Amazon SES SendBulkEmail at scale.
The process begins with this input:
[
"emailaddress@example.com",
"emailaddress1@example.com",
"emailaddress2@example.com",
"emailaddress3@example.com",
...
]
Hereās what we need to achieve:
- Create a valid object with parameters like so:
[
{
"Destination": {
"ToAddresses": [
"emailaddress@example.com"
]
}
},
...
]
While this task is relatively straightforward, we need to loop through each email address and add the payload as shown above.
- Divide the object containing email addresses into batches of 50 items (due to Amazon SES hard quotas). Even better, we can use the AWS Step Function intrinsic function like so States.ArrayPartition($.toAddressDestinations, 50)
- For each batch, call the Amazon SES SendBulkEmail with the correct parameter structure.
One thing that proved super useful was getting the parameter from outside the Map state. This is possible using ItemSelector like so:
{
"ContextIndex.$": "$$.Map.Item.Index",
"ContextValue.$": "$$.Map.Item.Value",
"toAddressDestinations.$": "$.toAddressDestinations",
"templateData.$": "$.templateData"
}
Finally, our workflow is done. Before wrapping everything up, we need to set up a proper error flow. In this case, if there are any errors, my workflow stops, and I get alerted via email.
See? A complete workflow without AWS Lambda functions!
Challenges
During the process of building the serverless AWS infrastructure, I encountered several challenges, but Iām glad I was able to complete the AWS Step Function workflow without the need for AWS Lambda functions.
Letās explore some of the challenges I faced (and no, I wonāt talk about the enormous time spent on HTML/CSS š):
- Unfortunately, there is no built-in utility in AWS Step Functions to enable marshalling and unmarshalling results to and from Amazon DynamoDB. This means you need to handle .S or .N every time.
- You can identify errors from SendBulkEmail by examining this object: $.BulkEmailEntryResults[ * ].Error. In my case, I used this object to list every error. If any were found, I raised an error, stopping the entire workflow.
- Flattening the DynamoDB output requires two steps. Unfortunately, itās not possible to accomplish this in a single, straightforward step.
- Itās not possible to use AWS Step Function intrinsic functions within the Choice state. Consequently, in many cases, a new Pass step needed to be added.
Metrics
Visualization of metrics is crucial, and SES offers two main capabilities:
- Metrics for tracking Bounces, Complaints, Sends, and more.
- Virtual Deliverability Manager (VDM) for Amazon SES.
The default metrics are fairly standard. Now, letās look into VDM functionality.
Virtual Deliverability Manager for Amazon SES
VDM provides a lot of metrics, including soft and hard bounce rates, open rates for emails, and more. While it lacks a āclick per linkā metric, which we had to build that ourselves. Apart from this, there arenāt many fancy or custom metrics available; however, the standard insights provided are fundamental for any newsletter.
Additionally, you can download a CSV file containing every single metric available in Amazon SES, which proves to be quite useful.
Total Pricing
Letās quickly summarize the pricing for Amazon SES and AWS Step Functions. Hereās a comparison:
Amazon SES:
- 1,000 emails: ~ 0.3$
- 10,000 emails: ~ 3$
- 1,000,000 emails: ~ 30$
AWS Step Functions:
- 1,000 emails: ~ 2,150 executions = free
- 10,000 emails: ~ 21,500 executions = ~ 0.5$
- 1,000,000 emails: ~ 2,150,000 executions = ~ 55$
ā ļø The price could potentially be reduced to 3$ for AWS Step Functions by incorporating the āCompute to bulk destinationā Map into the DynamoDB processing. This could be a notable improvement for the future.
Conclusion
Creating a serverless newsletter with Amazon SES and AWS Step Functions is a potent and cost-efficient solution. By leveraging SES for reliable email delivery and Step Functions for orchestration, weāve built a scalable system without traditional Lambda functions.
Well now you know how to create a serverless newsletter on AWS, the only thing I didnāt show you is how it turns out!
Check the results š https://cloudnature.net/newsletters/subscribe
Iāll post monthly, covering the latest AWS trends, community blog posts, and Cloud-related news. If youāre an AWS Cloud Architect, Cloud Engineer, DevOps, or AI/ML Developer, youāll find valuable content.
You can find the repository here: https://github.com/Depaa/newsletter-manager-template š.
If you enjoyed this article, please let me know in the comment section or send me a DM. Iām always happy to chat! āļø
Thank you so much for reading! š Keep an eye out for more AWS-related posts, and feel free to connect with me on LinkedIn š https://www.linkedin.com/in/matteo-depascale/.
References
- https://docs.aws.amazon.com/ses/
- https://middy.js.org/
- https://docs.sst.dev/
- https://www.mail-tester.com/
Disclaimer: opinions expressed are solely my own and do not express the views or opinions of my employer.
In Plain English š
Thank you for being a part of the In Plain English community! Before you go:
- Be sure to clap and follow the writer ļøšļøļø
- Follow us: X | LinkedIn | YouTube | Discord | Newsletter
- Visit our other platforms: Stackademic | CoFeed | Venture | Cubed
- More content at PlainEnglish.io