Tuesday 10 November 2015

CRDF antivirus test

I went to the CRDF web site. CRDF uses Virustotal for testing, so any caveats that apply to VirusTotal, will also apply to the CRDF results.

The very best product (based on today's chart) is showing a 62% success rate (all the others are 50% or worse), which means that two in five malware files are flagged as clean. I get about a dozen per day, so even if I used the best product available, it would be a rare day that I didn't get hit by malware.

I wouldn't use this test as a guide to the comparative capabilities of AV products, but I would take this as meaning that there aren't any AV products that are actually of any value.

I've been putting this point to the people who I knew from the old days, meaning the people I knew 20 years ago who are still in the AV industry. The answer has, mostly, been a deafening silence.

But there was one exception, who pointed out, correctly, that these results are taken from the command line scanning capabilities of the product, and don't allow for any benefits that products might have by virtue of being connected to the internet, and therefore having access to information from the product vendor's server that might improve their results.

I'm fairly sceptical about how much benefit that would give. Here's the problem.

Bad Person makes the malware. Today's malware isn't made by "kids having fun", which was what was happening in the old days (20 years ago), it's being made by people aiming to make a profit, so it's worth their while investing some effort in their project.  Bad Person then tries it against a few scanners; if it's flagged (for example, by the heuristic), then Bad Person changes the malware. And keeps changing it until the scanners pass it as clean. Because what's the point of blurting out malware that's already detectable?

Now Bad Person mass mails the malware to a million email addresses; it's pretty easy to get a million email addresses by parsing web pages for the "Mailto:". But Bad Person doesn't email the exact same file to each address; Bad Person makes each one a bit different, to make the AV job harder (the technical term for this is "server-side polymorphic"). The malware arrives at these addresses a few minutes later, because email is quick. Maybe some of the AV companies get a copy, realise that it's malware, and start working on it at once. They have to make detection, check that there's no false positives, hope that the copy they have isn't different from the other 999,999 copies, and now they can put their update on their server.

It's a race. While they're doing all that (and doing the same thing for all the other malware that gets emailed), I'm reading my email. It's not surprising that most of these emails won't be flagged by most products.

And this race is repeated at least a few dozen times per week (based on what arrives in my mailbox), and there's probably malware that isn't emailed to me, I have no idea what percentage of all malware is arriving at my mailbox.

We need a better solution than is currently on offer. I've already suggested how this could be done.


  1. Yes indeed you did and I want to take you out to your favourite restaurant and discuss it with you :)