My Email Setup
After many years of fighting with spam and disorganization, I have reached a smooth and trouble-free email configuration. I thought I would write it up and share it with others, particularly since I know so many people who struggle with spam.
Active Spam Killer
I discovered Active Spam Killer (ASK) when I sent email to Kirk McKusick, one of the founders of the BSD operating system. It sent me a confirmation message. I figured that if it was good enough for someone with as high an online profile as him, that it was worth trying. I have been quite pleased. Previously, I had messed around with Bayesian content filters, with disappointing results.
Active Spam Killer works on a simple whitelist mechanism. If the sender's email address is known to ASK, the message is allowed through. If the sender is new, the message is held in a queue and a confirmation message is sent back to the sender. If the sender responds to the confirmation message, the original email is delivered and the sender's address is remembered.
It has worked incredibly well. Though about a thousand pieces of spam are sent my way daily, very little of it gets through. Lately, it has been an average of about one every two days. These are usually messages that get confirmed through by the spammer, but these are infrequent because such spammers get caught rather easily. Meanwhile, I have never, as far as I know, failed to receive legitimate email from an individual. The only thing I have to keep an eye on are automated email messages from web sites and such.
I manage two lists, a whitelist and a blacklist. The whitelist started by me collecting together known sender addresses. ASK comes with a tool to generate a starting list by looking through your mailbox. However, it is no big deal if you miss a few, because they will get added later. Every few days, I will open up the list and examine addresses that have been automatically added. If they belong to spammers, I delete them from the whitelist and add them to the blacklist. If they are legitimate people who sent me messages, I leave them in the whitelist.
To give you an idea of how well this works, my blacklist currently has 156 entries after nine and a half months of using ASK. A spammer confirmation typically generates two entries (the From: address and the SMTP envelope sender address). So that's at least 78 messages, and probably less than 90. That's like 10 a month.
One weakness is in regard to automatically generated emails, because in such cases there is usually no one around to respond to the confirmation. For me, this happens most frequently when I buy something from an e-commerce site. It sends me an order confirmation, ASK sends it a confirmation which goes into the bit bucket, and I never see the order confirmation. I could probably live without the order confirmations, but they often include useful things like tracking numbers, so I prefer to get them. When I am about to order from a site, I will add the site's domain to my whitelist (before placing the order). That fails about a third of the time because the address it is sent from often has nothing to do with the web site's domain. In that case, I have to open up my junk folder and look for it. When I find it, I correct the address in my whitelist. Another situation in which this sort of thing happens is when I subscribe to an online communitity that wants to verify my email address.
Another weakness is the amount of "noise" that ends up in my mailbox. There are two kinds: undeliverable messages and confirmations to myself. If ASK sends a confirmation to a bad email address and it gets bounced back, ASK usually dumps the bounce and I never see it. However, there is a huge variety of software out there acting as a mail transport agent, and they often do not send bounce messages in a way that ASK can identify. They get past ASK, and I end up having to delete them. The other case is when spammers make email look like it came from me, which causes ASK to send the confirmation to me. ASK won't do this if it sees that it is my email address, but I have a whole lot of email addresses (root@somecustomerdomain.com and that sort of thing), and I haven't told ASK about them all. I may take some time to construct a couple of procmail recipes to reduce the amount of this noise, but I suspect it will always exist to some degree.
Active Spam Killer is written in python, and it gets installed on the mail server (which must presumably be running Unix or Linux).
spampurge
This is a flexible little utility that I wrote for managing the ASK queue directory and my junk folder. ASK puts things into the queue, but as far as I know, it never purges anything, so eventually it fills up all the available disk space.
I run spampurge hourly from cron, and it looks for a .spampurgerc file in everyone's home directory. The file contains a list of objects to purge, the type of object, and the length of time to retain things.
Active Spam Killer has a .ask/queue directory in my home directory, and every message it queues for confirmation gets stored there in an individual file. I instruct spampurge to delete anything in that directory that is older than a week (168 hours).
I have all my rejected messages stored in a junk folder. It is a regular mbox format file stored in the Mail directory, so I can access it using IMAP. Anything more than a day old gets trimmed out of there.
I also have a pesky dead.letter file that keeps appearing. I haven't bothered to research what causes mail to go into there, but it is always spam. I don't like seeing it my home directory, so I have spampurge delete it whenever it sees it.
Here is the contents of my .spampurgerc file:
dir .ask/queue 168 mbox Mail/junk 24 delete dead.letter
procmail
procmail is good for a lot of things, but I use it mainly to file incoming mail into folders. I primarily do this for mailing lists.
The cool thing is that procmail is designed with hooks for calling a mail filter program, and ASK is designed to be called by procmail. So, when I receive incoming mail, procmail gets executed by sendmail because of the contents of my .forward file:
"|exec /usr/local/bin/procmail"
And procmail executes ASK when it reaches this rule in my .procmailrc file:
:0 fW |ask.py --procmail --loglevel=4 --logfile=/home/mpearce/.ask.log :0 e junk
The first rule executes ASK as a filter, and the second rule stores the rejects into my junk folder.
IMAP-UW with SSL
There are basically two protocols for Mail User Agents to retrieve email from their mail server: POP3 and IMAP4. POP is the most common, but it is rather simplistic, intending the MUA to download all the email and delete it from the mail server. A bunch of problems arise if a user tries to use POP with two different computers. In such cases, IMAP is much better suited.
IMAP is designed to allow multiple computers or MUAs access to the same mail repository. It also allows a user to keep multiple mail folders on the server, all of which can be shared by all the MUAs. This is why I use IMAP. It can also be used as the back end for web-based MUAs, such as IMP.
By default, POP and IMAP conversations are not encrypted. This includes the user's password. Various mechanisms have been built to do authentication with POP and IMAP using non-cleartext passwords, but no standard has been broadly adopted.
I avoid the whole nightmare by encrypting the entire network communication socket with SSL. SSL is the same encryption protocol used with "secure" web servers (HTTPS). Not only is my password encrypted, but so is the email.
Thunderbird
I have gone to using Thunderbird on all my workstations. It is feature rich and stable. It is also multi-platform, so I can use it on Windows and FreeBSD, and I will be able to use it on MacOS when I get there.
iPhone
I have an iPhone, which comes with an email client. The client works relatively well. One drawback is that it shows me every IMAP folder on the server, not skipping subdirectories or anything. It can be tedious selecting a folder. Otherwise, it works rather well.
There still do not appear to be any good options for enabling push notifications for IMAP (or POP).
pine
(pine is not elm)
When I need to send or read email from my mail server, I use pine now. I have not become much of an expert with pine, but I can do what I need to with it. Also, since pine is provided along side IMAP-UW, it doesn't conflict with its internal storage.