Failing Me Softly with SPF

👓 3 minutes

TL;DR — The joys of the Sender Policy Framework and the (apparent) difficulties of implementing it correctly.

About a year ago, I started receiving some fairly odd email messages: some originating from my own domain (itsericwoodward.com), and many originating from my own email address!

Most of these emails were obvious spam (or were themselves the product of spam, such as failure messages from external domains where the spam had been sent), and I started panicking: had I been hacked?

As I dug into the issue, I discovered that no, I hadn’t been hacked, but that my domain was just blindly accepting requests to send email. That’s when I first learned about SPF (Sender Policy Framework), “an open standard specifying a technical method to prevent sender address forgery.” Basically, to use SPF, I (as the domain owner) simply had to add a single TXT record to my domain information that indicated what servers could actually send email from my domain, and all others would be rejected. So, after fiddling around with the format for a while, I found my hosts’s recommended record (name changed to protect the innocent):

v=spf1 a mx include:mail.example.com ~all

Needless to say, I added it to my domain record, and bam – overnight, nearly all of the spam stopped flowing.

So, I went about my merry way, figuring I had solved the issue. I even added this record to the other domains that I own, since it had worked so well with the first one.

But it turns out that I was wrong.

Although I stopped receiving messages the messages from myself, every now and then I’d get a failure notice about an email coming from one of my domains being unsendable. When I’d get these, I’d think, “huh, I thought I fixed that,” and then I’d forget about it and move on to something else.

Then, just the other day, I received an email from me, to (another) me, telling me that I had a voicemail from an international number (it even helpfully suggested that “You might want to check it when you get a chance.”). And I thought, “wait a minute, didn’t I already fix this?”.

So, this time, rather than forgetting about it, I actually took a moment to look at the header, and that’s when I saw this (again, name changed):

softfail (mail.example.com: domain of transitioning xxxxxx@example.com
  does not designate 192.0.2.1 as permitted sender)

So, I went back to the SPF site again, and that’s when I learned that the server could not only reply with “pass” and “fail”, but a whole cornucopia of messages:

Received-SPF: softfail (mybox.example.org: domain of transitioning
   myname@example.com does not designate 192.0.2.1 as permitted sender)

Received-SPF: neutral (mybox.example.org: 192.0.2.1 is neither permitted
   nor denied by domain of myname@example.com)

Received-SPF: none (mybox.example.org: domain of myname@example.com does
   not designate permitted sender hosts)

Received-SPF: permerror -extension:foo (mybox.example.org: domain of
   myname@example.com uses mechanism not recognized by this client)

Received-SPF: temperror (mybox.example.org: error in processing during
   lookup of myname@example.com: DNS timeout)

The next obvious question is “why?” (well, that and “what the hell does transitioning mean?”).

It turns out that my host either misinterpreted the SPF spec, or tried to protect their users from themselves. Apparently, the presence of the tilde (“~”) in a record indicates that the domain is in transition to SPF - it’s designed for large email providers and corporations to let them start transitioning to SPF without forcing hard failures on every email that originates from an unlisted server (the idea is that the email owner would collect these softfails, verify the IPs are valid, and then add them to the SPF record). Oops.

Fortunately, I now knew what to do to fix the problem: replace the tilde with a dash (“-”):

v=spf1 a mx include:mail.example.com -all

A few hours later, problem solved, and no more spam from my domain (so far).

Based on my reading, I’m not the only one that made this mistake, and since SPF has been around for a few years (and is implemented, in some for or another, on most corporate domains), I can only imagine how many other domains using an incomplete / incorrect implementation of it (based on the number of F*-buddy requests and Canadian pharmaceutical offers I still get, I’d imagine that number to be fairly high).

So, if you own one or more domains, please do everyone a favor and implement SPF for your email. And if you aren’t using a ton of different mail servers (wherein you might not be able to list them all in your TXT record), skip the tilde and go straight for the dash.

The internet will thank you for it.