January 3, 2007

In defense of Challenge-Response spam detection systems

Like a lot of people, I get a lot of spam. And in the past months, it has gotten a lot worse. On average, I get well over 1,000 spam a day, and that is after spamassassin has already processed it and deleted what it detects without me ever even seeing that. Of those that get through, Outlook puts most in the Junk E-email folder, leaving me about 100 a day to delete - intermixed with my 50 or so legitimate emails a day.

In terms of actual time lost, it is not great. Maybe 2 seconds per email to delete or 2*100=~3 minutes plus a couple of minutes to look through the junked email to salvage good ones (of which I typically find about 1 per day) for a total of maybe 5 minutes a day.

But in terms of mind share and detraction, this is huge. It means that I am continuously distracted all day long by the dregs of society - pornography, rampant commercialism, and fraud. This is the worst kind of distraction, not only taking my mind away from my flow of concentration, but doing so in a way that I do my best to avoid in every other aspect of my life, and that I would not even consider letting my 7 year old daughter have access to.

So, after giving up on all the standard solutions to spam, I signed up for SpamArrest, a commercial "challenge-response" spam detection system. This works by requiring everyone that wants to send you email to first follow a link to a website and prove they are human by reading a word in a warped image and typing it (i.e., a CAPTCHA). The reason this approach works is that:
  • Each sender only has to do this once for me. The system remembers that person for the future.
  • I can preload the system with all of my contacts and anyone I've sent email to in the past so that everyone I already communicate with won't have to validate themselves, and won't know I am using this system.
  • New people that send me email have to use this system once, and legitmate senders are usually willing to go through this step.
  • I can authenticate senders unlikely to do this (like various large e-commerce sites).
  • I can let email lists through by setting them up indivdually.
  • Spammers that send me email are almost never willing to go through this step, and so I never see their email. The reason that spammers aren't willing to do that is because they are computer software and can't, or because they are human and don't want to spend the time. In fact, most spam is sent by "spambots" which are other people's computers hijacked for the sole purpose of sending spam. This email is sent with forged email return addresses, so they never even receive the request for validation.

So, if this is such a panacea, why isn't everyone using it? Well for one thing, you have to pay for it (about $3/month). But lots of people think this approach is a bad idea in principle, and have been arguing against it. However, while I agree that it does have problems, it is better than any current alternative, and I'm not going to wait around suffering while I wait for better solutions. So, let me respond to one complaint about challenge-response systems. I'll summarize the complaints and respond here.

Concern #1. Spammers will forge mail to me with someone else's return address thus sending my challenge to the poor forgee's email box.

Looking at the actual spam I receive, the vast, vast majority has false return addresses. And of the legitimate ones, most of those very likely come from spambots running on machines that have been infected. The owner of those machines have a lot more serious problems than deleting my challenge to them. In fact, it may tip them off to the fact that they are infected. And of the few third party legitimate emailers who get my unwanted challenges, I apologize. But that is still a tiny, tiny fraction of the total spam in the world. I'll gladly stop when there are better solutions. And I won't get mad if I occasionally get unwanted challenges from others (which I do, and which is a tiny, tiny minority of the total spam I receive).

Concern #2. If a challenge-response person emails me, then both our systems will challenge each other, generating even more email traffic.

So what. We each accept each other's challenges and we're done. We only have to do this once per person. And again, this one-time extra 2 emails is so tiny in the wide world of spam, that it is a totally irrelevant argument.

Concern #3. Challenge-response systems are easy to defeat since all someone has to do is forge the From address as someone that I already trust.

This is the easiest to respond to. Yes, in theory this is true, but in practice, spammers don't know who I trust. And in the past 4 days, I have received 4,491 emails of which 165 have been classified as good. Of those, about 30 were spam, but all of those spam were sent through mailing lists that I trust, not from forged From addresses. This does bring up a legitimate problem which is that popular mailing lists may become targetted as spoofed return addresses. But again, in practice, this has not happened yet. So I'm not going to avoid using a system because it theoretically might not work at some point in the future.

The bottom line is that challenge-response systems are not perfect, and probably won't work as well if everyone uses them. But for now, they work much, much better than anything else short of a human spam deleter (now there's a good business opportunity!). And if they stop working better than alternatives, then I'll switch to whatever works better.

December 5, 2006

Kojo Nnamdi "Computer Guy" answers questions

I had the pleasure of being a "Computer Guy" on WAMU's Kojo Nnamdi show along with Allison Druin and John Gilroy. We got to answer a number of email questions on the air, but here are some other ones with my answers.

It is of course, good to see answers, but I do this also to point out the sorry state of affairs of the computer industry as this is just way too many problems for every day computer users to have.



How do I print a listing of the files in a directory?

Alternatively, you can still do it from the command line with the following steps. Run the command shell by pressing Windows-R, and entering "cmd" in the popup dialog and click ok. Change to the directory you want to list with the "cd" command (i.e., do something like "cd c:\video"). List the directory and send the output to a text file with the command "dir > listing.txt". This runs the "dir" command and sends the output to the specified text file. Then go back to Windows Explorer, open the newly created "listing.txt" file in notepad or Word or any other text editing program and print from there.

Subject: question for the Computer Guys

I have Verizon DSL service. When I open Internet Explorer to my home page washingtonpost.com, I get all of these popup boxes saying "Runtime Error." The boxes contain information such as "Object Expected" or "Null Object." and asks if I want to want to debug. Three questions - what causes Runtime Errors? Does debugging fill your computer with spyware? And why is only IE affected by such errors?

If this happens occasionally, it could be due to a temporary problem with the website in question. But if it happens frequently, then it is probably due to some broken plugin for Internet Explorer. That is, you may have installed some software that modified Internet Explorer and did so in a bad way. For IE 7, try Tools->Manage Add-ons->Enable or Disable add-ons, and try disabling stuff. If you are using IE6, try updating to IE7 with Microsoft Update.


Subject: Vista

Hi Guys!

I am one of the few people left on the planet who is still running Windows ME, and have been looking forward to getting a new computer when Vista comes out. Should I buy it right away, or wait to see how it performs?

Thanks,

In general, unless you enjoy living on the "bleeding edge", and the excitement and inevitable problems that go along with it, I recommend waiting a few months after any major technical product is launched. In this case, it probably means waiting until after SP1 (Service Pack 1) comes out to fix the first round of bugs, probably late spring.

Subject: wireless internet security

My home wireless router indicates that I have "security-enabled wireless network (WPA). What's the difference between WPA and WEP wireless security? Is one better/stronger? Is it safe to buy and bank on-line with WPA? Thanks.

WPA is the newer one, and I recommend it over WEP whenever you have the chance.

Subject: Shared Outlook Contact and Calendar Files

Hi, I've paid for a Dell server to be set up running Microsoft Small Business Server software for my small, home-based business. Despite many calls (and lots of paid invoices), cannot share either Contacts or Calendar files from within Outlook. They tried to set up a separate outlook folder, but the server won't default to it, they say, unless we wipe the server and I pay to set the whole thing up all over again. Any advice on how to get these programs to share over a small network of 2-3 desktops, please?

Many thanks,

I'm afraid I don't know how to help you here - but it sounds like whoever you paid to set up the system for you did not do their job properly. If you've tried to get them finish the job and can't get them to do so, then don't pay them and pay someone else to start over. If you already have paid them, then ask for your money back. If they won't, then you've learned a hard lesson about not paying for services before they are completed. Sorry.

Subject: Computer Guys Question

I got a virus on my desktop PC. It was cleaned out but in the process it lost its IP address and now I cant get onto the internet without that, is there a way to restore it? Thanks

I'd start by turning off your cable modem and computer and any other hardware and then turning it all on again (starting with the modem and router and powering up your computer last). If you still can't get it to work, try calling your internet service provider.

Subject: computer guys question

Is there a side-by-side comparison of processors available ... a chart, etc? How do AMD, Pentium HT, and dual processors stack-up? What direction do I go with my next purchase keeping Vista in mind, and knowing that I am now doing more photo and home video work.

There are loads of comparisons out there, but I'm not familiar with any one that I can suggest. Try googling "processor comparison" to get started.

Subject: Transfer audio tape cassette music to PC

Is there an easy way to transfer my old audio tapes to my Windows XP PC so I can hear my music on CD's?

David Pogue discussed this very issue recently, so I suggest reading his description.

Subject: older computers and external hard drives

Help! We have a 6-year old Sony Vaio PC, 60 GB, Pentium 4 processor. We use it for basic applications like word processing, Internet surfing and email. Since we began storing picture and music files the computer has slowed dramatically and now constantly tells us it has no room left on the C drive. I say we need to just buy a new computer so that we can store and manipulate all our digital images and music. Thrifty Hubby says we just need to buy an external hard drive and move our picture and music files there. Please help advise! Our happy marriage may depend on it!

You are both right (how's that for marriage counseling). Buying a new hard drive probably will delay the inevitable, and improve the situation. But using a 6 year old computer for today's pictures and music probably just isn't up to the task. I sure hope you are backing up those valuable pictures and music. Every day. Automatically.

Subject: Firefox

I switched to Firefox because of a corrupted IE6, that wouldn't even start. Tried any number of fixes, including the oxymoronic "Microsoft Help". Downloaded IE7 but the problem lives on. I love Firefox, however now I can't get Windows updates, because surprise, surprise, Updates
will only download thru Internet Explorer. Am I missing a tool feature or preference in Firefox, that will allow a it to talk to Microsoft?

This sounds bad. Your only solution may be to reinstall Windows (and then reinstall your applications). You can do so without affecting your personal data. But definitely back up first to be safe.

Wii @ Home

After all the hype and violence surrounding XBox 360 and PS3 with graphics to die for and matching games, I was thrilled to finally get my hands on the family friendly Wii.

And impressively, it is essentially all it is cracked up to be. The easy-to-learn controller really is easy to learn. Some games, like tennis don't use any buttons at all. The full body motion really does get kids moving and interacting like never before with a video game. But most importantly, it engages the whole family. I can promise you I have never before heard the words "maybe we should get one" from wife before anywhere near the words "video game".

But it isn't all roses. The games seem to use very limited information about body movement. So, in tennis, you can't move the player from side to side - and there seems to be only a single bit of speed. I'm afraid this just won't stay fun for that long. But that's ok as the free intro game - as long as others go deeper.

And the family friendliness is sometimes pretty superficial. "Rayman Raving Rabbids", for instance, is a first-person shooter that trains kids for the worst in videogames - dressed up as cute rabbits with shooting plungers.

And the risk of whacking your sister in the head with a wiimote is real. I caught it happen on video in the first 10 minutes of use. (Fortunately, no one was hurt.)



(Thanks to colleague Bill Pugh and family for hosting the wiiParty.)

November 10, 2006

Adobe and Apple give me no respect

Somehow I would have thought it would be obvious by now, but major software vendors still abuse their privileges when installing their software on my machine. This comes in various forms, but often includes adding buttons in the top part of my Start menu, in my quick launch area, system tray, on my desktop, and even embedding themselves in other applications - all without warning, and sometimes without the ability to undo their actions. Two recent offenders are Adobe and Apple.

Adobe Acrobat Professional 7 on Windows XP is the worst offender. Given the nature of the product, it is reasonable to integrate with some other programs, and I am happy to be able to generate PDFs directly from Word. And the fact that they chose to offer that capability by adding a button to my Outlook toolbar is reasonable. However what is not reasonable is that there is no way to disable that feature. They configured their button to be always on, no matter what. When you try to configure the toolbar to remove the button, it is the only one greyed out - meaning that they think converting to PDF is more important than actually sending an email!



The reason I care so much is that because it is an always-on button, if you make the toolbar small (like I do to avoid having toolbars take over my screen), then it is the one that stays visible, and the buttons I actually want (Send and Accounts) disappear. The only solution I found is to edit the registry to disable the PDF plugin to Outlook. So Adobe has forced me to disable their product so it doesn't drive me crazy. Nice job Adobe - now I can be mad at you every single time I send an email.

The second offender is Apple. They nicely make it easy to offer an update to QuickTime and iTunes, but every time they update, they re-add buttons to my desktop and to my QuickLaunch bar. Do they really think that since I removed them the first time, I'll want them the second time? And do they really think that a minor security update should give them the opportunity to get in my face? Nice job Apple - now I can be mad at you too every time you offer me an update.