Why is Email So Hard?

As an Internet services consultant, I continuously deal with email. Next to the Web, email is one of the Internet services that is considered essential. Can you imagine running a business without email? Me either. But, increasingly so, running an email server is much more complicated than running a web server.

Here’s the deal. A web server basically stands by itself. There’s a set of protocols (http, https) and languages (HTML, more or less) that have become more and more standard since the beginning of the web. The fascinating history of web browsers uncovers complexities in designing web pages for multiple browsers when their features diverge, and when quirks or bugs need special workarounds. But as long as your users have a compatible browser, and they know where to find your web pages, keeping it all working is fairly straightforward.

On the other hand, email servers are designed to work as part of what we can call a federated network. The “Fediverse” has gotten plenty of attention lately in the context of microblogging (think Mastodon, Pleora, Threads, etc.), but email is a far older example of separate servers operated by independent entities that are designed to talk together. For your users to get emails, they not only need a compatible client program, you also need to ensure that your server plays nice with every other email server in the world that hosts other users that your users want to communicate with.

Email Protocols

There are basically two protocols that are involved in email. Alternatives exist and are used frequently, but I’ll focus on the two most common. IMAP is the protocol that connects a user to their mail server; and SMTP is the protocol that mail servers use to talk to other mail servers.

IMAP

When you want to check your mail, you open your mail program. This might be a program on your computer or phone, such as Thunderbird, Outlook, or Mail. At least as popular these days is a mail program in a web app, such as Gmail, Roundcube, or Rainloop. In all cases, this program needs to interface with your mail server in order to retrieve your email. Almost all of these programs use the IMAP protocol. IMAP stands for Internet Message Access Protocol, and it provides a standard mechanism for any mail program to talk to any mail server. (Just like HTTP provides the standard for web browsers to talk to web servers!) This is great, because it means you are not locked into a specific app that only works with your ISP; and once you find an app that you like, you can take it with you if you move to another ISP or open a new mail account. This interoperatbility is part of what makes email the versatile communications platform that it is.

There are a few exceptions, like I mentioned earlier. Gmail actually provides its own app to connect to Gmail servers; while they do support IMAP (more or less), some features of their mail system are only available when you use their proprietary app. In the same way, Microsoft Exchange servers provide mail services that are designed to work with their proprietary Exchange email apps. More on Gmail later, but even these special proprietary mail systems are still able to communicate with other, standard, mail systems behind the scenes.

SMTP

When you send email, it’s almost guaranteed that it ends up moving across the Internet via SMTP, the Simple Mail Transfer Protocol. This is how most mail servers talk to each other. Just like if you send a physical letter through the post system, it may make several hops and transfers before it gets to its destination, when you send an email message it may get sent from one email server to another before it finally lands in the mailbox of the intended recipient.

That may sound more complicated than it needs to be. Why not just send the email directly from your computer to the computer of the person you are communicating with? Well, because they might be using a completely different software stack, with different configurations. Common protocols like SMTP allow different systems to communicate with each other in an organized way. Without federation, you would be locked into specific service providers. If different email programs couldn’t talk to each other, then only gmail users could send messages to other gmail users, and you would need a umich.edu email address to communicate with someone else with a umich.edu address. Additionally, the flexibility of SMTP allows for improved deliverability if a temporary network issue prevents immediate delivery of your message.

So, What’s the Problem?

There are many problems with email. Email in its current form is built on protocols that were designed in the 1980s, in a time and place when things like security, privacy, and encryption were not particularly important. At its beginning, email was not designed with any mechanism for verifying that a sender was who they said they were, and there was no built-in encryption to keep third parties from reading the contents of a message. While additional features and workarounds have been added to make email more usable for our modern needs, an email protocol designed today would probably look very different.

But I want to focus on one particular problem, one that has almost nothing to do with the protocols and everything to do with the corporate forces that direct who is allowed to use email and for what purposes.

The Spam Problem

While Hormel would no doubt prefer we use a different term, any conversation about email is likely to include mention of spam. In the context of email, spam has come to refer to any unwanted email. Often, mail server operators try to be more specific about the nature of unwanted email. Other related terms are unsolicited bulk email (UBE), phishing scams, unsoliciting marketing emails, or junk messages for the purposes of pranks (trolling) or harassment.

It is so easy to send email messages that unwanted email has always been a significant problem. So it’s no surprise that email service providers have spent tons of time and developed many many tools and techniques for blocking unwanted email. One popular tool, and one I use, is rspamd, an open-source mail filtering system that, briefly, uses a scoring system in part based on Bayesian algorithms (in other words, old school AI) to separate desirable mail from undesirable mail.

Another common strategy is for mail administrators to block particular mail servers that send unwanted emails, and to share their block lists so other adminstrators can block the same servers. Since most senders of unsolicited email send a lot of unsolicited email, it’s usually easy to spot the problem servers. This technique also puts pressure on administrators to reign in their users and keep them from sending unwanted email: if they let one or two users send a bunch of unwanted mail, their server might end up on a block list and none of their users will be able to send mail.

Strategies like these have been around for years, if not decades. Server administrators have had plenty of time to get very good at separating “good” email from “bad” email. But spam is still a problem. Why? Well, capitalism, my friend.

The Gmail Problem

Let me start by saying I like Gmail. I don’t mean to pick on Gmail here, or Google in general, but I use them as an example of a broader problem of consolidation (and its seemingly inevitable consequence, as Cory Doctorow calls it, enshittification).

Google revolutionized email when they created Gmail. It was a leap forward from previous mail programs, doing away with folders (so antiquated!) and replacing them with tags; incorporating algorithmic filtering and searching to help sort through hundreds of messages; creating a responsive web app that looked basically the same no matter where you were (as long as you could get online); offering a free service that actually worked pretty well; and virtually eliminating the concept of ever deleting a message. Perhaps just as important, Gmail is tightly knit into their popular ecosystem of apps including Calendar, Drive, Docs, and Sheets.

Because it was so good, soon everyone wanted a Gmail.com mailbox. According to one source, by 2024 Gmail has over 1.8 million users and handles over 30% of daily emails worldwide. Roughly 22% of the world’s population uses Gmail.

Of course, Gmail is far from the only big mail provider. Other large providers include AT&T (which owns Yahoo, SBC Global, Bell South, Ameritech, Prodigy, and more), Microsoft, and AOL. Additionally, a whole industry has developed just around sending mail, including dominant companies like SendGrid, Mailchimp, AWS/SES, and Mailgun.

Consolidated Corporate Control

With so much market share, companies like Google have an outsized influence on how email works. Not only do they drive mail client features like filtering and tagging, but they pressure all other mail server administrators to operate in a certain way. In other words, if you want to send email to a gmail.com address, you have to play by Google’s rules. This isn’t necessarily a problem by itself, but if a large company’s rules are arbitrary or damaging, everyone suffers.

This concentration means that the communications of tens of millions of people are dependent on the invisible, inscrutable and inexplicable decisions of the administrators for a small coterie of giant firms.

Cory Doctorow, Pluralistic.net

Consolidated markets also hurt individual users. If you rely on Gmail for your mail, Google (like most companies) has every right to discontinue your service at any time, for any reason. If Google arbitrarily decides they don’t want to do business with you anymore, the consequences could be painful.

To make matters worse, consolidation doesn’t just make it harder for individuals and small businesses to send email. It also makes it relatively easier for big players to get around the restrictions. Big mass mailing companies are able to get away with sending spam because they can pressure other mail administrators into keeping their servers off of block lists. The monied interest of larger companies can provide a lot of extra leverage.

How To Fix It

Email is a minor annoyance for anyone who uses it (which is basically everyone), but it is especially annoying for anyone who relies on email to communicate with lots of people. Email gets even more challenging when you send a lot of it, even if you send only to people who have specifically asked to receive your messages.

Unfortunately, there’s no easy solution to a systemic problem. But there are things you can do as an individual or an organization to reduce the pain that you feel.

• Don’t rely on a large commercial email service, especially a “free” one, for your organization’s email accounts. Even as an individual, using your own domain for email, instead of @gmail.com or @aol.com, means that you can easily move to a new ISP if you have problems.
• Use a mail program on your own computer, instead of relying on your ISP’s webmail. This means that you automatically have a backup of all your email in case you lose Internet access or your ISP shuts down.
• If your organization relies on email for daily operation, have your organization host its own email server. That means you aren’t dependent on other companies or services for your own internal needs. (Most importantly, it means that your administrator can guarantee delivery to your organization’s own mailboxes, so you never have to worry about Gmail misclassifying important messages as junk.)
• While sending email to arbitrary people will always come with some friction, running your own server gives you more control over the details. You can pay to play with one of the big providers, but putting all your eggs in one basket may not always be your best choice.

In a federated network, you can’t control everyone else’s servers. That is, you can’t force someone else (an individual or another email administrator) to accept your emails. But, if you choose to, you can always control your own mailbox. Don’t give away that control to a big corporation, especially if they are not looking out for your best interests.

“Free” always sounds good, but that doesn’t mean it is.