Everyone here talking about no email clients, IMAP and AWS restrictions is missing the point. Although you could use this as a personal account, or as a company. This fully automated system is a perfect ground layer to a number of applications with a very cleaver API.
Say you are a small startup, and you will need some email functionality? Give your web server access to writing to outbox and you can send email. Use the sender with + rules and you get replies with customerId, threadId etc. Need to show email interactions in your service UI? Subscribe to the notification channels. Need customer support platform, it is almost built-in.
If you don't see value in this solution, you probably never tried to setup infrastructure to receive email. In fact, at $work, I developed a very similar architecture and it is running flawlessly for 15 months or so. We just don't do all this git+lambda craziness, we just had already setup EC2 that we added some routes.
As somebody who has been an early engineer at many startups, I would never use this in almost any case because a managed solution is just better. If I can pay somebody else to do something hard with zero maintenance so I can focus on shipping a good product, I’m going to do that every time.
It really depends on what you are doing. Even though I have a similar setup, we are still outsourcing customer support with a third party for now. But when you do need the data from emails, I would expect this solution to be a better choice.
Fiddling with outsourced APIs has a lock-in factor as well as not guaranteeing time-save if you don't have someone on the team already familiar with the third party product. I've certainly burned myself excessive outsourcing as well.
True. Why not though? It really depends on your software choices. Security might be one. But with just writing to S3 you get to have the JSON stored as sent, and might be easier to write tests against, since in dev you can write the same JSON to FS and write some assertions... really up to whomever is implementing
It only store email on S3, that mean you cannot read email with your email client.
Seem thing with outgoing, you upload email to s3 in a certain format to be send out, not smtp protocol
This is more like an API for sending and receiving. You cannot say generate SMTP credential and drop in your mail client.
Also, one of the limitation with AWS SES is you cannot sent FROM or RETURN-PATH to anything, even in production mode, you can only set frm or return-path to verified domain.
I have an internal AWS SES/Lambda stacks that I used for email forwarding(only receiving) which use SES to accept email, trigger a lambda and use SES to send a copy to a personal inbox. Plan to cleanup it a bit and release
Back in the days I remember storing emails in similar way and reading using an email client. It was called ”maildir” [1].
Emails were stored as files in folder structure. I think this goes quite nicely with the unix ideas as you could use many standard tools to work with the messages.
I do something similar to OP [0] - and currently forward to another provider (single address, but that's only the 'envelope' To; so it doesn't matter, I preserve the original whatever.to.address).
That's mostly an artefact of transitioning from that provider to my own on AWS, but I stopped working on it for some time and never continued. My plan was/still is to write a client that can read 'Just a Bunch of .Eml' format, and then I can point that at S3 (or via S3FS or whatever). Everything that exists for mailbox, maildir, and what-have-you assumes you need it to act as an MTA (I think that's right) and do the actual receiving too; and is difficult/impossible to stop that and say 'no, here's a directory full of email, just be a reader'.
Longer-term plan/idle thought is to have a JMAP server on Lambda. (IMAP isn't possible, because it has to look like/be HTTP to reach the Lambda.)
But, are you sure you can set "From" header to anything(even for email/domain that aren't verified with SES). So that on receiver side(gmail for example) it will show the original sender.
In my account(in production mode, not being sandbox), I couldn't. When I attempted to set "From" header(or "Return-Path") I got these error
554 Message rejected: Email address is not verified. The following identities failed the check in region US-WEST-2: can-we-overwrite-this@my-domain.com, Test random <raindom@gmail.com>...
By envelope-from (whatever @ verified with SES), do you mean the `MAIL FROM` command in SMTP protocol or do you mean the `header FROM` in the email header?
I don't know how `send_raw` work under the hood in Rust, but without a way to set `from` header. how do recipient(say an @gmail.com) address show the original from too?
Did I miss something here? It would be great if somehow the email is forward as it's in so it appear in my inbox and all the information(header) is retain.
> just wrap the original email in an 'envelope'
It would be great if you can help clear my mind on how to do that. Say I got an email as this:
FROM: original-from
DKIM: dkim sign
Return-Path: etc
Other header:
Body of email appear here
How can I use AWS SES to forward that email as-is to my @gmail.com account for example while still keep in the original header intact(so DKIM/DMARC still work. SPF will be break but it's fine for me)
That's really weird though. I'm really curious to see how did you made it work because I want to make it works too.
As I understand, an mail client, like gmail.com web UI, use FROM header to show who send this email(not the MAIL FROM, which apparently is envelop FROM address).
So my questions: without ability to changing the FROM header in email, how can your mail client display original sender? because the original sender is outside of your domain.
So I have a domain: `acme.com`. Someone, let's say from OJFord@gmail.com send an email to `acme.com`. And I want it to forward to kureikain@gmail.com using AWS SES? But obviously, and I can confirm by trying `smtp.send_raw`, I can only set "FROM: "header to a verified address. But in my case, the "From" is "OJford@gmail.com"...
> As I understand, an mail client, like gmail.com web UI, use FROM header to show who send this email(not the MAIL FROM, which apparently is envelop FROM address).
Correct.
> So my questions: without ability to changing the FROM header in email, how can your mail client display original sender? because the original sender is outside of your domain.
We don't need to change it? We leave FROM exactly as it is on the original email (if we change anything DKIM will fail, since the signature won't match).
Conceptually it's like receiving a secret sealed letter with anti-tampering mechanisms built in, leaving it alone, putting the whole thing in your own envelope and sending it on to the intended recipient who will check the anti-tampering mechanisms.
At the SMTP level I believe the way this works is you connect, say you have MAIL FROM yourself, the relayer, that is RCPT TO your destination Gmail address. After that the entire headers and contents are exactly as received to SES.
> I can only set "FROM: "header to a verified address.
That's not the case, you only need to verify the 'envelope' MAIL FROM address, e.g. relayer@acme.com
> Also, I think MAIL FROM/RCPT TO(Envelop header) rarely being used to show on a mail client.
Yes that's right, but that's what I want. In 'normal' use it looks the same in the client as if the original email had been sent straight there (i.e. in your example, your Gmail set-up to receive for acme.com). If you want to though most clients (incl. Gmail 3 time I used it) do have a menu option somewhere to show you the full raw email, or just the headers, which is helpful for debugging.
> I really hope to make it works. So anyone can send email to my domain, and I can retain their original FROM. like the example above.
Yes, that's exactly how I use it. SES receives for acme.com, dumps the raw email into S3, triggers Lambda. Lambda connects to Gmail (in your case) SMTP server, says it has mail from something-arbitrary@acme.com for kureikain@gmail.com; and then sends the email on from S3.
Are you trying to implement it at SMTP level yourself, or using some lib? I used 'lettre' (rust), and to be honest I haven't checked exactly what commands it sends, so for all I know there could be more to it, a 'I am sending on behalf of someone else' or something.
I download and run your Rust code. And apparently it's same as others. You cannot set the FROM header to any value, therefore cannot forward the email as it's by retaining the original header.
thread 'main' panicked at 'Failed to send test: Permanent(Response { code: Code { severity: PermanentNegativeCompletion, category: MailSystem, detail: Four }, message: ["Message rejected: Email address is not verified. The following identities failed the check in region US-WEST-2: random@gmail.com"] })', src/main.rs:41:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
We're saying the same thing? I said no, you can't modify FROM, but you don't want to (for my/these purposes anyway).
What you can do, and what the code does (with values from env) is set the envelope addresses.
The email ('letter') inside is still from the original sender, and to my original receiver. It's just 'repackaged' in an 'envelope' from my relay (some arbitrary but SES verified address) to wherever I want it to end up (e.g. Gmail in your case).
It looks like you tried to send 'from' Gmail, which doesn't make sense, [edit: I suppose you could verify just your Gmail address in SES, if you really wanted to use that as the 'envelope from'/`MAIL FROM`] in this setup you want to send it on from Lambda (SES having received it) to Gmail.
You could probably cobble together a POP server on top of S3 pretty easily (e.g. using something like https://github.com/femto113/node-pop3-emitter which I threw together a couple years ago for a related purpose). I'm not sure if you can get it to work in Lambda though because the POP3 protocol needs sockets, but you could definitely get it to work in a simple container in Fargate (so still "serverless").
This argument is so tiring... if we all agreed to change the word “serverless” to idk “xirverous” it would have a pretty clear meaning and clear distinctions from “server” and “local to the machine”. Xirverous technologies are an innovative, new, unique, and interrelated class of technologies. Thank you for your consideration.
When you uploaded your PHP script, you uploaded it to a single server, and it ran on a single server. Unless your provider were doing some kind of load balancing (doubt it).
With serverless you upload your 'script' to the 'cloud' and then it will run on any number of servers, usually assigned to one when it dry-runs. Then it'll unload from the server afterwards. And therefore unless you have a bottleneck (like a database server) then you can essentially scale as far as the cloud will let you.
At least with Mediatemple’s shared hosting, they had a “grid” service that would host your site on a cluster of servers and claimed to be able to handle large bursts of traffic.
It worked because each page request ran a complete cgi php script to generate the page view and then terminated. There was no long running server.
This isn’t a contradiction. “Server” is an overloaded term and the “server” in “serverless” refers to a different meaning of “server” than the “email server” bit. “Serverless” means there is no physical or virtual hosts to manage, you just supply the request handler i.e., the email server. It's amusing because it sounds like a contradiction, but it's not one.
"means there is no physical or virtual hosts to manage"
We're just swapping one type of management for another, really. Becoming proficient in all the permutations of AWS stack, for example, to
* make sure you don't get overbilled,
* have appropriate memory/limit resources on your lambdas,
* have correct backup/redundancy on s3,
* make sure all your IAM policies are correct and not allowing malicious actors, etc.
and more... it's ... just a different set of things to worry about. Yay - I don't have a 'server' that can go down. I now just have to make sure I don't get billed $87k because I forgot to click the correct button.
It's not that hard to configure a lambda to run in just as limited a way as a physical machine.
Not having to worry about MCEs or disk failure patching makes the concerns less like a "different set" and more like a subset of things to worry about relative to managing servers.
I know most of this stuff is really well trod-over, but from your comment I think one'd get the impression that people are switching just because of a trend or something, not because there's an actual layer of management they're paying to have outsourced.
(I would acknowledge that view of it being a subset rather than a different set is invalid once you're debugging performance at a gritty level, where cold starts etc. etc. introduce their own equivalent layer of complication, but most people doing most things never need to)
How many times did you update your Node.js AWS Lambdas or GCP Cloud Functions because of the Linux kernel CVE of the week? You didn't because all you're responsible for is your few lines of Node.js logic that kept on scaling and humming along. The cloud vendor cares for the rest.
Just because you didn't doesn't mean they did and you didn't actually have any vulnerabilities. There is no such provable attestation of security in serverless model.
> Lambda provides support for these runtimes by continuously scanning for and deploying compatible updates and security patches, and by performing other runtime maintenance activity.
- Scales to zero (zero calls to your endpoints, $0/mo)
- Scales up automagically to match load
- You don't do anything to provision/patch servers (e.g. FaaS)
Classic PHP shared hosts don't meet this definition because:
- You paid a flat monthly hosting fee (e.g. Dreamhost)
- You only scaled to 1 node (need more, too bad)
True, you didn't usually need to mess with PHP.ini or Apache server installs and configs, but you still didn't get the same benefits as modern serverless.
I wonder if you could, for reasons, make a set-up where you upload your PHP files to an S3 bucket, and then have a API Gateway or Lambda which basically just serves the files from the bucket, and if they are PHP, executes them with a Lambda PHP runtime.
So you hit /index.php -> checks S3 bucket for index.php -> executes that with lambda and gets string of response -> returns it to original lambda -> serves up to client.
I'm a little confused about why you're responding to me. I think I agree with everything you're saying, but I'm not sure where the comparison with PHP hosting is coming from. Are you sure you responded to the right thread?
It's indistinguishable from a regular web hosting. You just upload script to some ftp and don't manage anything either. You just configure links to other services like DNS, or DB if that's needed.
It's just now the same thing is done via some proprietary/custom cli tools and/or APIs not via a standard protocol and a trusty old clients that are already in your distro, and it's branded differently.
APIs are good for some uses. That's usually lacking with hosting providers. Other than auto-scaling I can also easily bankrupt myself with if I make a mistake, I don't see much qualitative difference. Only in the details.
Yes it is. The "server" in "serverless" means no need to low-level manage/provision a server (computer) and its lifecycle. While "server" in "A serverless email server" means actually "service" as in "A serverless email service on AWS using S3 and SES". Which makes much more sense.
The linked repo lets you have an "email server" just like GSuite gives you a "serverless email server".
I don't understand. You insist it's a contradiction but then go on to reiterate my own argument which demonstrates that it is not, in fact, a contradiction but rather using the same term ("server") to refer to two distinct concepts ("computer" vs "service")?
This would make a lot of sense if the de-facto restaurant experience involved you cooking your own eggs, especially if the egg-cooking experience was notoriously laborious and expensive.
If we swap "egg" and "cooking" with "VM" (or "bare metal" if it suits you better) and "maintaining", suddenly "serverless" doesn't sound so ridiculous.
I disagree with your assessment, email is a service and not a server. A service can run on a server, or ephemerally as in this article (which still has servers underneath it, you're just not running them). S3 and SES are services running on servers.
Those aren't mutually exclusive, which is my whole point. One definition of "server" is "a service in a network". Another definition is "a computer in a network". So you can have a server running on a server without contradiction. If the user isn't responsible for managing the underlying computer, then you have a serverless server.
> which still has servers underneath it, you're just not running them
Of course, "serverless" means you don't have to worry about the underlying servers, not that they don't exist. Perhaps you were only clarifying and not nerd-sniping, but this particular nit is so boring and predictable in every serverless thread.
> One definition of "server" is "a service in a network"
Can you link to something where this is a common accepted definition? I'm a systems engineer by trade, talk to a bajillion people about all sort of things and we just don't call a "service in a network" a server in parlance.
I was not nerd-sniping and could care less about serverless as a term, I'm specifically talking about calling a "service" a "server" in this chat. I feel this is presenting something as accepted definition which does not match my experience in the field.
> Can you link to something where this is a common accepted definition? I'm a systems engineer by trade, talk to a bajillion people about all sort of things and we just don't call a "service in a network" a server in parlance.
This seems to be a common point of confusion for people who aren't native English speakers. Oxford English Dictionary defines "server" as:
> a computer or computer program which manages access to a centralized resource or service in a network.
Similarly, a quick bit of Googling turned up [this][1] which isn't authoritative, but indicates that "server" can mean either hardware, VM, or software services.
> This seems to be a common point of confusion for people who aren't native English speakers
Where did this come from and what value does it have, other than being condescending to non-native English speakers? I am a native speaker and we're having a discussion in my native tongue about words in my native language about work I do as a profession.
> Similarly, a quick bit of Googling turned up [this][1]
I do not accept Stackoverflow as an authoritative source for anything. Useful? Yes, great for finding random solutions to random problems. Authoritative source on terminology used in the industry I work? Nah.
I'm surprised you need a source for this. Literally half of the server software out there refers to itself as "<thing> server".
Apache [1] "The Number One HTTP Server On The Internet". This is not referring to the machine hosting, it's referring to the software that you run to provide a service.
Where did this come from and what value does it have, other than being condescending to non-native English speakers?
I don't think he was being condescending, so there's no need to be offended on behalf of other people.
I think he was suggesting that part of the linguistic confusion comes from how the tech industry has become so global that different words and phrases are exchanged between cultures, but within the tech sphere.
For example, it's not uncommon to hear the phrase "Do the needful" in places like Seattle, even though the phrase originated elsewhere and was imported by tech workers.
I do not accept Stackoverflow as an authoritative source for anything.
> Where did this come from and what value does it have, other than being condescending to non-native English speakers?
What is condescending? I'm observing that it's a common problem among non-native English speakers. It seems like you're taking offense on behalf of others and unduly so.
> I am a native speaker and we're having a discussion in my native tongue about words in my native language about work I do as a profession.
This is a common idiom among English-speaking IT professionals. If you're not familiar, that's fine. Now you know.
> I do not accept Stackoverflow as an authoritative source for anything
In a minute of Googling, I found several random sources on the Internet that indicate that the term is overloaded precisely as I described. One of those sources was the Oxford English Dictionary. I think that suffices to demonstrate that this is a common idiom, but I can't force you to be persuaded. ¯\_(ツ)_/¯
Nah. +4 for the thread after adding up the down/up votes. I dipped out, the conversation held no further value to continue to me especially after being told "well it's an idiom you just don't understand". No minds will be changed, no positive outcome will happen from continuing.
This makes no sense (too abstract of a statement), but here let's try this:
$ man systemd | head -4 | tail -1
systemd, init - systemd system and service manager
An email daemon (Postfix, Exim, Dovecot, UW-IMAP, Sendmail, etc.) are services running on a server. An "email server" would be a server running a daemon to provide email service in this example.
Man pages can be tricky as the content can vary distro by distro (how up to date is your copy of man-pages, what changes have happened, etc.) - this technique is pretty portable to extract the synopsis of a man page specifically due to the format it uses (most times I'd use like grep -A3 -B2 foo /some/file, e.g.).
I literally had one of the heads of a crucial flight system project loudly exclaim to me in front of the senior managers. "It's serverless - there is no server!"
To which I honestly, just kept my mouth shut and stood aside.
To which I honestly, just kept my mouth shut and stood aside.
I can never keep my mouth shut in these situations. Sometimes it leads to promotions, but I suspect it puts a ceiling on how high I can climb the corporate ladder.
Spent the first 10 years of my IT career driving towards greater control over the systems we managed, because "the reason the service is down is that someone else (usually a vendor) didn't do their job" was never an excuse. The last 5-10 years I've found myself ceding more and more control to cloud service vendors (like Microsoft). Less work for me ("raise a premium case"). Less stress too ("they're working on it, nothing for us to do"). Now I finding myself hoping I'll be retired before the pendulum swings back again.
I just figured there was some "short term" misunderstanding. As always I looked like an idiot but hey in a good place with good people youll all just move onto the next topic and know the details will work themselves out regardless who is right.
We went through this when people first started calling quadcopters "drones".
It's just a term used to describe a server configuration that has different billing/operational characteristics than traditional always-on.
And like the quadcopter-drone, I hope everyone losing their shit over this can either accept "serverless" or submit and market a new term that conveys this difference soon.
A cgi-bin is as highly available as the web infrastructure hosting it, which as the cgi-bin author you don't need to know anything about.
A cgi-bin cannot "crash" because there is nothing long-running to crash. It is only run on demand.
I think it is easy to make the case that CGI was the proto-serverless api and that the serverless offerings we have today are an evolution of the CGI approach.
And cgi-bin was really just an evolution of inetd (or ucspi-tcp if you worshipped at the altar of djb). You didn't used to have to worry about binding to ports or long running processes. Something would just send you a message on stdin and you replied on stdout.
It can be, and not necessarily. `cgi-bin` can be as stateless as you want it to be. Just as stateless as lambdas, or other forms of "serverless" functions.
And thus, it can be distributed as well as any other stateless function.
I ran an e-mail service with 2m users. The web frontend was written in C++ (no, I don't want to go back, but we had a pretty nice custom framework) all served up as a CGI, behind a load balancer.
If our code crashed, the next request would just get served as normal. If Apache crashed or a frontend crashed, another one would just take its place. High availability cgi-bin is trivial.
"Serverless" is a ridiculous stupid term. You'd think it meant something like Photoshop, some system that doesn't rely a server, something that lives locally. Nope. It's just a server that someone else manages 90% of.
No we call those a more succinct term: local processes. We call "serverless" for processes running not locally that do not require standing up a server that you care about for, are billed by usage only ($0 for none, infinity for infinite), and you are not concerned where or how it is running. Are there computers listening for requests somewhere that someone is managing that is doing this? Yes. But from my frame of reference it is a service, not a server. I know a lot of people on here get hung up on it, but it is the term and it is not going to go away.
Serverless has become a stupid marketing term, just like Facebook hijacked the words “Friend” and “Like”.
I look forward to true serverless software - that is either peer to peer or uses a distributed back-end with encryption. Now THAT is the future!
Serverless just means “we spin up a server for you”, similarly to how “the cloud” is just a euphemism for “extreme centralization of hosting myriad clients under the control of one company”.
It's more than that, slightly. A "serverless" application doesn't have a specific piece of hardware supporting it except in the context of a single operation. The hardware is all abstracted away and allocated from a pool on demand. I suppose it comes from "serverless architecture", where there are just other distributed services (not specific servers, virtual or otherwise) executing your code.
I agree it's not a term well suited to its popular use, but such is life. Commercial interests are constantly mucking up the language to suit their own ends. E.g. Discord is abusing the term "server" to mean a virtual meeting space. "Organic" w.r.t. food has a really contrived official criteria, unlike in chemistry. Regular people muck up the language in frustrating ways, too. See "literally".
I'm a little bit disappointed by the solution of this article.
They start with:
"This stack was created out of frustration due to the fact that to this day there's no easy way to have a full email server without the overhead of installing and configuring all servers needed to handle incoming and outgoing messages."
Ok, so the point is that it is too complicated/frustrating to deploy an email server.
But look at their solution in "what will deploy"... in the graph there are over 40 entities! In the following they list 13 different aws components needed.
And all of that not even really doing the core feature of email sending/receiving that is provided by SES.
Maybe you can deploy it with a single cloudformation recipe, but then it is so so many systems that you will have to watch and check for issues, quota, policies, usage billing and configuration.
And you are not even in control, you have to go lick aws ass for easing the restrictions on the number of emails sent and to who. Plus there are probably a number of drawbacks and hidden limits that are not exposed to you by aws and that you will discover the hard way.
For exemple, looking at the description I'm wondering, how do you do coherent backups at a random time? Knowing the quality of service of aws api, what do you do when there are random crash, network errors and co in the middle of the pipelines/lambdas?
In my opinion this is a good exemple of an over-engineered solution.
There are a class of engineers who have very little Linux experience, but use AWS extensively. For them, "Setting up a server" is always going to be more complex and frustrating than using the platform that they are familiar with, even when the AWS setup they come up with to replace a single server is way more complex to an "outsider".
Personally, I don't think there is an excuse to not learn Linux. It's usually the right tool for the job, and you can use tooling like ansible and AWS instances to get your stack cloud based, stateless, automated, yada yada buzzword . But I can't really blame people for using the tools they are comfortable with.
Reading it I get that AWS now by default restricts email by
- 200 mails per day max. 1 per second.
- Can only send mail to verified addresses. Not sure how they verify though.
- Can only send mail from the domain you own and verified.
All these are good steps. Guess that will gradually take AWS IPs out of all those mail blacklists.
And while we are on the topic, I find IP based mail blacklist services such as BRBL obsolete and possibly harmful. Now that we have domain verification services like DKIM SPF etc. Blacklist the domain. Not ip addresses as most institutions don't own their IPs like in the past. Most of all are hosted, and that IP might get assigned to someone else a few days later.
These restrictions are only for the SES sandbox mode. You fill in a form to get out of sandbox mode and then you can send many more per day (still only 14/second or so).
You only verify addresses you want to send from (for testing you might verify one to try sending to as per the sandbox rules). You wouldn't get customers to verify their addresses here, as that would let you send email on their behalf.
All very good info that as far as I know is all pretty unique to SES regarding the default/sandbox vs the upgraded production capabilities. The set up makes a lot of sense to prevent abuse and it works just fine out of the box for dev purposes. The production version also operates way more how you'd actually expect an email service to work.
All that said, I don't know of any other services that have this "production" upgrade that is needed other than maybe just the general service limits (which have always seemed more of a "protect you from yourself" situation) so it can be a touch confusing.
Sandbox mode is also great for low-volume internal mail like notifications from cron.
My guess is that the main argument for sandbox mode is that it requires you to explicitly say you don’t spam and will handle reports, giving them carte blanche to yank your account if you do. Given how slimy email marketing is I suspect someone at AWS didn’t want to deal with people lying to their support people all of the time.
And once you are out of sandbox mode, it scales pretty well. We regularly send millions of mails in 24hr periods. Some gotchas come in when you need to set up DKIM/etc and other providers may need to handle mail for the same domain.
> I find IP based mail blacklist services such as BRBL obsolete and possibly harmful. Now that we have domain verification services like DKIM SPF etc. Blacklist the domain.
SPF and DKIM require opt-in by the sending domain, of which the majority of non-commercial non-US outgoing MTAs do not do. IP-based blocklists, including Spamhaus PSBL Mailspike etc, are all valuable to catch IP addresses that should not be directly connecting to your MTA. I agree that signing up for an outgoing MTA service such as Sendgrid makes some IP-based checks obsolete but that's not where the majority of spam is coming from. Botnets continue to be the number one source of junk email and will likely continue as we add more and more insecure doorbells, garage openers, refrigerators, etc, to our home networks.
> Botnets continue to be the number one source of junk email and will likely continue as we add more and more insecure doorbells, garage openers, refrigerators, etc, to our home networks.
Surely that's an argument against blocking IPs?
Unless I suppose you argue they're predominantly in homes; and homes are predominantly not running (intentional, well-behaved) email servers, so sorry-not-sorry those who are.
Yes. The days of running a legitimate MTA at home are unfortunately over. Many US-based consumer ISPs block port 25 entirely except for connections to their own MTAs, but for the ones who don't, very very many of those IP addresses wind up on policy-based blacklists -- eg "this network is a bunch of consumer addresses and should never be connecting directly to an MTA"
Running from home IP sucks for other reasons too, like non-static IP addresses, or lack of public IPv4/use of CGNAT.
I certainly would not risk my emails being delivered to someone else's computer just because IP address changed, and DNS still points to the previous one.
But having the server and data at home is still possible via VPN.
> Guess that will gradually take AWS IPs out of all those mail blacklists.
Are you thinking of EC2? SES has never had a problem with this in my experience.
Regarding verification, you can verify any email address where you can receive a link long enough to click on it. Domain validation is only needed if you want to send mail from arbitrary addresses without that step.
Individual SES IP's have been [temporarily] blacklisted before. We hit this issue earlier this year with one spam filter/service, and you can find plenty of past examples in the SES forums.
Sorry, I should have said significant problems. If you send email on anything like a large-scale basis you’ll get on some kind of blacklist at some time – all you need is one user sending email to someone with a twitchy spam reporting reflex. What I was thinking about, though, is that it’s usually transient – e.g. out of hundreds of thousands of emails I’ve never had one bounce that way instead of being delayed, which is better then self-hosting on a long-time class B over a similar timeframe.
SES is pretty good at monitoring your email usage, such that it maintains a "reputation" score calculated by monitoring the hard bounce rate generated by your account. The premise being that high bounce rates (AWS define this as >5%) strongly indicate you're not managing your email recipient list properly i.e. sending untargeted SPAM.
"I find IP based mail blacklist services such as BRBL obsolete and possibly harmful"
Fully agree to this. I've always wonder why spam filters don't place more importance on domain reputation. So many senders runs on the same IP for deliveries and it's really silly to blatantly block someone based on IP.
For example, $89.95/mo is the starting price to get a dedicated IP for sending with Sendgrid.
If the smtp server is allowing spam to be sent through it, then it is not trustworthy. It doesn't matter which domain it was sending on behalf of. Though the real reason is that we've only had spf records and what not for a relatively short amount of time, so most systems are designed from the perspective that you can't trust the domain a message is claiming to be from.
IPs are still a precious commodity, hence the threat of being listed by a high quality blocklist like Spamhaus provides an effective degree of encouragement for senders to clean up their act.
Domain blocklists also exist, but the stick isn’t as sharp, because domains are a dime a dozen.
Meh. Something like 95% of mail servers wont work with IPv6 last I checked. Source: I helped build and maintain an MTA that moved over a trillion emails last year.
They've been limiting max sends per day for a long time. I set up an account around 4-5 years ago and they had a limit back then. I forgot the exact amount but it was 200 or less.
But, they do make it pretty easy to get the limit raised. I filled out a form and about a day later they bumped my limit to 50,000 per day and I think I only requested 10k per day. The form required basic information like providing a reason why I wanted a higher limit and a link to my main site where I was collecting emails from. It seemed to be human reviewed.
They must just give 50k as the baseline. I asked for the minimum, which I think was 200, and they gave me 50k. I was surprised because my plan for handling bounced email was “by hand.” Lol.
>> I find IP based mail server blacklist services such as BRBL as obsolete and possibly harmful. Now that we have domain verification services like DKIM SPF etc. Blacklist the domain. Not the ip
Agree with you 100%. I want to run a mail server on my AWS server and not use Yahoo so that Verizon doesnt snoop on my mail and insert creepy ads that seem like emails. But this is impossible as my mail would get dropped into a blackhole.
We are building a similar serverless no-code inbox (with SES and S3) not for outgoing emails, but to only receive them [0].
Our service doesn't require a user to sign-up with any existing identity (like email or phone number), and so, when they make payments, we send the invoices from the payments-processor to an email (corresponding to the customer-id) on our subdomain that goes to SES which promptly plonks it into an S3 bucket set up to life-cycle it out after 60 days, encrypting it with keys associated with customer-id. This encryption serves us well when the customer chooses to rotate / delete their pseudo-anonymous identity (and the keys along with it), as then all data in our systems associated with their id (the archived emails, for example) is essentially tombstoned.
The reason we prefer serverless over other solutions is not because of its scale or low-cost, but high-availability.
From another side, this looks much more complex, that getting an email service with catch-all address and use rules/filters to do the same.
I'd programmatically manage rules/filters and folders, rather than this entire system.
All email providers I have used on scale (Google, Microsoft, Rackspace) support this.
I'd still have a usual email service with SMTP and IMAP, rather than incompatible custom JSON APIs and unknown failure modes.
What is really interesting is cost on scale. Because if it is cheaper to send/receive 100M emails though this, than it's worth trying. However this is exactly the part not covered at all. And if you are sending not a lot of emails, maybe pay $50 to major email provider and spend your time on something more important?
There's another much more important limits on AWS SES that the article doesn't mention if you try to do this for normal email, the 10MB message size limit, which translates to about a 6MB maximum attachment size.
It works fine for transactional mail and most newsletters, but quite a few users expect to be email larger files than that.
When the documentation says "this may cost you money" what they really mean is "this will cost you a lot of money the moment your email address gets onto any spam lists". The number of lambda invocations will go through the roof, and even if you're able to catch 90% of the spam before dumping messages into S3, the cost of putting a message into a bucket will make your bill skyrocket periodically as spammers temporarily find their way past SES's built-in filters, which happens every so often.
Assuming a default message size of ~10kbytes the SES pricing is about $6 for the connections plus the chunks. Your lambda pricing is negligible and the S3 pricing gets you another dollar or so.
But you're operating at the wrong scale-- 30k messages to a domain can happen in an hour by a spammer doing a directory attack. The point is that this solution is extremely open to abuse that happens all day every day over SMTP. Without a fixed-cost firewall to reject messages and only allow clean messages through to this solution, you are going to have escalating costs.
I dunno... I run my own email server and it feels to me like the giant gap in the ease of getting your own email server (or services) up and running that you can connect to a web app still hasn't closed much here, especially for me because I don't use AWS and I don't want to spend the time to learn.
My experience with 3rd party email services is they're also a pita to use and that can become huge when they merge or get bought up by a competing service or die, or raise their prices.
With the web browser we can configure very simple emails that they can send with a link to click via the user's email client but as a web app developer what I really need is to be able to send a html email that way.
Maybe web browser and email app makers could add a way to do that by letting apps request permissions and users approve it for a specific web app. That would really make life easier for me and my apps users.
This is great. I tried a similar approach but couldn't use it as a transactional email server because of this (taken from OP's readme):
> By default, you can't send emails to unverified addresses. If you'd like to be able to send (as opposed to just receiving), you'll need to reach out to AWS to remove this limitation from your account.
I reached out AWS to remove this limitation but they refused my request 3 times claiming that my use case would impact the deliverability of their service.
AWS is not wrong. The instant that you stand up a service that allows anyone to send email, spammers will take advantage of it and spam the world. This ruins your reputation and you get blocked everywhere.
I'm not sure what the point is, since this solution doesn't give you an IMAP server to connect to with your email client. Also the solution seems a bit complex with all those CodePipelines and CodeBuilds. I think all you really need is one or two Lambda functions if you're just going to store stuff in S3.
We use exactly this solution internally at $work to handle our customer support, inbound sales, and other 'generic' group email that gets read by software and eventually gets to humans. It all goes through a separate inbound path and domain vs. our corporate email domain. Our outbound email from all of our apps goes through sendmail locally on the app servers and then out via SES.
All you'd really need is a file explorer that works with S3. Create a new file in the TMP folder, and it gets sent out. View received emails in the appropriate folders. It would also be trivial to setup an app or web form for creating the JSON file and uploading to S3.
> This flow was designed to take advantage of the S3 trigger system and break each action into a small Lambda.
The email flows from S3 => Lambda => SES so the underlying design seems to prohibit sending of emails directly from SES.
It sounds like SES could still be used directly, but you then need to add in authorization and user management if you're accessing it externally.
It's a cool project and I can envision it being used for internal tooling for automated email notifications or something of that nature. Just doesn't seem practical for users.
Sure, but pretty much every other email option out there has protocol support. Without that, I think it's a stretch to call this an email server. (Though there are a few that are only accessible via their client, such as Hey)
> This stack was created out of frustration due to the fact that to this day there's no easy way to have a full email server without the overhead of installing and configuring all servers needed to handle incoming and outgoing messages.
Looking at diagram, I'm not sure Postfix + Cyrus is more complicated than what you've got here.
Congrats though, I'm sure this will help a lot of people wanting to manage mail themselves. Just not sure how you're going to get mail accepted by others from constantly changing (and possibly blacklisted) Amazon IPs without using trusted smart hosts.
> Just not sure how you're going to get mail accepted by others from constantly changing (and possibly blacklisted) Amazon IPs without using trusted smart hosts.
Are you thinking about EC2 public IP addresses here? That service uses a different IP range than SES, which you can verify by comparing https://ip-ranges.amazonaws.com/ip-ranges.json to `dig amazonses.com TXT`. In my experience, we haven't had any issues with SES emails getting blocked, but you must set up DMARC properly and respond to bounces quickly.
The actual email sending / receiving part is handled by Amazon SES here. Similarly you could use Sendgrid in that place. This project is a wrapper around those. It also doesn't aim to provide an email inbox - this is for programmatic handling of email.
Now the interesting part to me is using the email address local part to create an email address hierarchy upon receipt.
This project stores incoming emails to S3 and the '+' in the email address indicates subfolders from left to right.
That's a very creative way to organize emails for processing.
So they can say it's their service.
Using somebody else's email service isn't their "serviceless".
Of course there is a question of where you draw the line.
But IMHO, using somebody's email service is that line here.
Otherwise it's not far from using gmail or aws WorkMail for that matter.
I feel like the current methods for spam filtration are getting so many false positives that it makes email unnecessarily difficult for users with good intentions.
I also don't know how much of this is unethical behaviour by the big email providers, forcing you to sign up for their cloud email solution instead of doing your own.
It sure would be nice to be able to run one's own email server without having to stress over whether the recipient got your email.
I'm confused, does this use different IP addresses constantly? How is this mailserver able to send email without an IP based reputation? Is it just that Amazon is so giant other mega-corp email providers are wary to blacklist their IP range?
If so this is a wonderful way to send spam without ever worrying about IP reputation. I imagine it will be abused and then blacklisted quickly.
The domain signing with DKIM still doesn't get you a valid IP reputation. It may help to gradually build one if it's the same IP over and over, but it's still a brand new IP suddenly sending mail from $domain. If it weren't an amazon IP it'd certainly get refused or at least marked as spam and hidden after acceptance.
Just another example of one set of rules for human people and another for corporate people. Either pledge yourself to a corporate feudal lord (amazon here) or suffer.
>$1 per active CodePipeline (must run at least once a month to be considered active)
I'm not too versed in how aws services work, but how many active codepipelines are you expected to have running every month? A few paragraphs up it says that there are 3x CodePipelines in total, but they might not be all active every month. If all of them are active, then that would come out to $3/month, which seems a bit pricey considering you can get an actual hosted email plan for $5[1]. If you're looking to save money you can go even lower[2][3]
I was hoping this would include a lambda set up to respond to incoming requests on ports 25/465/587, but all the SMTP parts (both server/inbound and client/outbound) are being handled by Amazon SES, not by the serverless functions.
It's not possible. I wanted to do that with mine (similar thing to OP) - not actually for SMTP but for IMAP or even POP3, but you can't with Lambda - you can only reach it (from outside AWS) with HTTP(S).
(I use a Lambda to forward to another provider currently, and use their client. I plan to write my own to replace that; ultimately I'd just like to run a JMAP server on a Lambda, that you could do.)
I use this myself for forwarding emails and signing up for random sites. Pretty easy to set up and 0 overhead to maintain. One of these days I'll set up a ui around it to send emails. Having both set up and maintained an email server "the old fashioned way" and using this, the over head on this is way less than postfix et al. This only goes for receiving email though, sending email with this is a biiit clunky, and doesn't support attachments (yet). If you have an aws account and a spare domain give this project a shot! It has a surprising amount of utility.
If you only want forwarding, you could look at https://www.nearlyfreespeech.net/services/email . You'd have to do a bit of math to figure out the total price (depending on if you want/need DNS there to make it work), but it is _really_ cheap and easy.
I built something on those lines, although way simpler in terms of functionalities and resources used.
All I wanted was to receive/send emails from a custom address (@acme.com) using Gmail, but I didn't want to pay Google Suite for that.
So I used SES+S3+Lambda to receive emails and forward then to a Gmail alias (email+acme@gmail.com) so I can attach a filter on them using a label. And for sending emails using the address, I simply authenticated it on Gmail using the "Send mail as" option using AWS SES SMTP credentials.
This is awesome & I totally intend to play with it :D
Do you have any plans / any ideas on feasibility to plop an imap server on top of the s3 store? I have really wanted a serverless aws solution and this moves the needle on work I did not wanna do so I’m reinterested again. Something like an s3 object change -> dynamodb index might work well for latency concerns and keep the serverless spirit alive. I just have such a starting problem on “building/learning imap intricacies when it may never work”.
This is a very misleading title. It is not serverless. It uses SES which stands for Simple Email Service, which is provided by (of course) servers. It's just that they are servers that Amazon maintains for you. It is no more "serverless" than any other hosted email provider.
This is not to say that this is without value. Having your emails end up in an S3 bucket is tremendously handy. But it's not serverless.
That depends on what you mean. If you mean, "Is it possible to have SMTP without a server?" the answer is no, by definition. SMTP is a client-server protocol. The same is true of IMAP and POP.
If you mean, "Is it possible to have something that has an email-like UI/UX without a server" the answer is yes. But it won't interoperate with what we call email today, because that is fundamentally based on SMTP.
In fact, the original email was serverless because it only worked on a single machine. That protocol is actually still in use today. Cron jobs error logging message of last resort is to send email to root@localhost, which is done without any networking at all.
Great initiative but I see one major concern: Spam filtering. Most of the big mail providers provide spam mail filtering out of the box without which your inboxes are bombarded with spam. This can be probably solved by spam checking by adding additional lambda, now you need more compute per mail, translating to more cost per mail, which makes it easy for malicious parties to send mail bombs and bankrupt you.
I was trying to set up Amazon Cognito (an authentication service) recently which basically needs a verified email address to send users emails for user verification and password reset, and I was shocked that Amazon has no basic email hosting service. Of course it’s easy enough to work around for my use case, but I was surprised nonetheless.
I saw that, but it was like $5/user/month and geared toward employee email rather than automation (my use case is my hobby projects for which my goal is <$5/month in total).
Note that using the + address "trick" in email addresses is silly: everyone knows it and can strip it out with a simple regex. It doesn't accomplish the main things that it exists to do, which are to a) make it non-obvious what your username for a site is and b) make it obvious where a spammer got your email.
The main thing it exists to do is to route email to folders or special handlers. It has nothing to do with spam or obfuscation, and predates the concept of spam.
This is really interesting! Currently trying to solve a similar problem thing myself, so having a look through to learn from your solution.
I'd recommend having a look at AWS CDK to manage the AWS resource creation, is much nicer than raw CloudFormation: https://github.com/aws/aws-cdk/
For my solution to this I've also then got a frontend ontop of my backend to setup view mailboxes, email list per mailbox, email viewer (read/unread status), and email sending. I'm doing the same with S3/SES/Lambda + DynamoDB for email sending/recieving/processing + email metadata in DynamoDB. Mine is currently WIP and needs some real auth (using a URL API key, not the most secure...) + email replying + tidying up. Will definitely open source my solution and post to HN when ready.
Yeah potentially I could extract the relevant CDK parts of my project out and combine with the code here to produce what you're asking for. I'll try and set aside some time and have a look.
Totally cool project! I can't quite figure out where in the repo the code is, or maybe this repo just contains the CF configs? If so a link to the lambdas etc. would be useful.
Something link that could be useful as disaster recovery action. I case of problems with the Mail infrastructure redirect the DNS to something like that and incoming e-mails won't get lost in the meantime. The standby cost for something like that is much lower then a independent secondary e-mail system.
Why redirect anything? You can have backup MX records. This is already a solved problem. And backup system can be quite lightweight, if it only needs to store emails for the time primary is down and then push them to primary once it's up again.
Everyone here talking about no email clients, IMAP and AWS restrictions is missing the point. Although you could use this as a personal account, or as a company. This fully automated system is a perfect ground layer to a number of applications with a very cleaver API.
Say you are a small startup, and you will need some email functionality? Give your web server access to writing to outbox and you can send email. Use the sender with + rules and you get replies with customerId, threadId etc. Need to show email interactions in your service UI? Subscribe to the notification channels. Need customer support platform, it is almost built-in.
If you don't see value in this solution, you probably never tried to setup infrastructure to receive email. In fact, at $work, I developed a very similar architecture and it is running flawlessly for 15 months or so. We just don't do all this git+lambda craziness, we just had already setup EC2 that we added some routes.