By Howard Wen
When most of us get email offering questionable herbal alternatives to Viagra or dubiously low prices on Adobe software, we simply delete it, having accepted long ago that receiving at least some unsolicited email comes with the price of using the Internet. But for Daniel Quinlan, it motivates him to figure out how to stop it -- for not just his sake but everybody else's. It's his job: He works as an anti-spam architect for an email security provider. And his paid work also carries over to his contributions to SpamAssassin, of which he currently chairs this free software's Project Management Committee.
Supported by the Apache Software Foundation, this server-side spam filter is one of the most widely-deployed programs in the open-source world. It's readily available for free, and is also implemented in many commercial packages. SpamAssassin is pretty much the standard in spam-filtering.
Quinlan has written many of the spam-filtering rules for SpamAssassin, most of its current DNSBL (DNS BlockList) implementation, and a lot of its HTML rendering code. "I generally work on whatever area seems interesting or is annoying me the most. I know some parts of the code better than others, and tend to focus more on those," sums up the 31-year-old, who resides in the San Francisco Bay area, California, USA.
He spoke with us about the general state of spam-fighting today.
OSDir.com: So which side is winning the war on spam right now -- yours or the spammers'?
Quinlan: It really depends on the user. If a good solution is installed, the anti-spam community is doing pretty well, but an astounding number of users have little or no spam filtering.
OSDir.com: When you mean by "user," do you mean the end-user who receives their email? Or the network administrator who should be maintaining anti-spam measures on the server side?
Quinlan: I mean end users. It's a good idea for network administrators to maintain anti-spam measures on the server, though, and the best solutions combine server-side and user-side measures. SpamAssassin can be run on a server using network and local rules plus user training.
OSDir.com: So how much of the anti-spam fighting responsibility would you say is the end user's email client software's verses the network's?
Quinlan: "Responsibility" is not quite the word I would use. If you don't mind deleting spam manually, that's your prerogative, but don't complain about it. If your ISP doesn't do a good job fighting spam, then switch ISPs or install your own anti-spam software. There are a lot of choices out there.
OSDir.com: Today, what would you say is the toughest thing about fighting spam, from the perspective of developing anti-spam software?
Quinlan: It's either the continually evolving nature of spam, or the fact there is so much broken and poorly-designed email software out there.
OSDir.com: What's the most effective anti-spam technology that SpamAssassin uses right now?
Quinlan: I think network rules are the most effective single technology, in particular, the URI rules that use SURBL, looking for spammer domains in Web links.
OSDir.com: Conversely, over the course of SpamAssassin's development, what has proved to be the least effective?
Quinlan: Any technique that tries to identify "good" mail without authentication backing it up, or some form of personalized training. It worked well for a while, but it's definitely not an effective technique today.
Updated for clarity: Quinlan: Any technique that tries to identify "good" mail with neither authentication backing it up nor some form of personalized training.
OSDir.com: In what ways have spamming techniques evolved over the past couple of years, which have made it a challenge for software like SpamAssassin? How has SpamAssassin been thwarted by spammers?
Quinlan: I'm not sure any single spamming technique bears mention, although the technique of spammers writing viruses to turn hosts into spam-spewing zombies has definitely helped spam remain a major problem.
The greater challenge is that the new techniques never stop coming. It's possible spammers will eventually run out of tricks, but it definitely hasn't happened yet. Most techniques backfire fairly in the long run, and make it more obvious that a message is spam.
A lot of people say spammers are forcing anti-spam solutions to evolve and keep up, but the reverse is just as true.
OSDir.com: What's the craziest/toughest spamming scheme that the SpamAssassin team has encountered and dealt with?
Quinlan: That would probably be advance fee fraud, also known as "Nigerian" or "419" scams. These messages are often literally sent individually to each recipient, mutating each time, by scammers typically located somewhere in West Africa. Because they often are sent in low volume, and almost every one is somewhat different, they are a bit tricky to catch.
OSDir.com: How has open source affected the evolution of SpamAssassin? Do you feel the open source model has led to more effective anti-spam development, compared to a commercial anti-spam project?
Quinlan: It definitely has. Almost every successful commercial anti-spam product is using the wide-spectrum filtering technique pioneered by SpamAssassin, and a large number actually use SpamAssassin under the covers.
OSDir.com: Has the open source nature of the SpamAssassin project made it vulnerable to spammers in some way -- spammers using this to figure out and exploit the program's weaknesses?
Quinlan: I'm sure some spammers look at our code, but the end effect is about the same as with closed source. To beat closed-source spam filters, all you need to do is install the filter somewhere or get an account at the ISP, then you just keep an eye on whether your spam is getting through.
Also, much of our filtering relies on stuff not in the source code: user training via Bayes, network rules like SURBL for URI blocking, various DNS blocklists, and message checksum systems like DCC.
To put it another way, closed source hasn't exactly protected closed-source programs from other types of security problems.
OSDir.com: What do you predict spammers have up their sleeve in the near-future?
Quinlan: Most spammers are already breaking the law, so there are few limits on what they'll try. My guess is they are going to become more sophisticated in mixing their spam in with regular email.
Anti-spammers are already aware of spam zombies being relayed through incoming MX servers. Since many sites use the same host for both incoming and outgoing mail, it's easy to find the MX. Zombie software digs out the SMTP smarthost from the mail client configuration, grabs user password information for SMTP AUTH and POP accounts, etc..
Ultimately, I think that once a spammer controls a system via a virus, they will start directly controlling the user's mail client (Outlook, Mozilla, etc.) when the machine is idle to send out spams that will look exactly like a message the user might have composed (headers, signature, etc.), except for the content of the message, and the fact that it will be sent to thousands of people.
OSDir.com: So you're saying the fight against spam will move more from the network side to the end-user's own inbox? At that point, will SpamAssassin become, essentially, irrelevant as an anti-spam tool?
Quinlan: Sorry, that's not what I was saying. Current spamware run on zombie machines -- those end-user boxes that have been exploited by viruses, includes an internal SMTP client that does most of the sending work.
I think smarter zombie software will begin to read settings out of the user's mail program, or even use the mail program itself to send the spam. Some viruses already send virus payloads this way, of course. Anyway, there's no reason SpamAssassin would be unable to catch those messages.
OSDir.com: What kind of technologies are in development or consideration for future releases of SpamAssassin?
Quinlan: One of the most interesting new technologies is email authentication. We already support SPF [Sender Policy Framework], and our 3.1 release will also include DomainKeys support. Future releases are likely to use both methods as the basis for white-listing of known "good" domains.
The beauty of the SpamAssassin approach is that as new technologies come along, we can easily adapt to include them.
OSDir.com: Do you think it's ever possible that we'll achieve a spam-free Internet?
Quinlan: It depends on how you define "spam-free." If you mean that nobody is sending spam, posting blog spam, sending spam over chat networks, etc. then I think the chances are rather slim. If you mean that most people will rarely see [email] spam, then I think it's possible. In that scenario, anti-spam measures become a permanent fixture like anti-virus software. I strongly doubt that spam will completely vanish, though.
OSDir.com: Does the SpamAssassin project need financial assistance, other than from the Apache Software Foundation, to do a better job?
Quinlan: Financial contributions to the Apache Software Foundation always help. Most of the costs to the foundation are infrastructure-related, so any contributions to the ASF will directly help SpamAssassin.
OSDir.com: How much of your time have you committed to fighting spam? What keeps you motivated?
Quinlan: Given that it's my job, I spend most of my working hours (and much of my play computer time as well) working on anti-spam stuff and fighting other email threats. It's pretty easy to stay motivated because the problem is very interesting and rapidly evolving. Working on the problem is also a very good coping strategy for all the cruft I get in my inbox.
Howard Wen is a freelance writer who has contributed frequently to O'Reilly Network and written for Salon.com, Playboy.com, and Wired, among others.