Thursday 25 February 2016

Antivirus product testing

25 years ago, AV product testing was really difficult. You had to assemble a zoo of viruses (and how would you do that?), check that each of them really were viruses (some zoos I saw had a lot of simply rubbish files), then see how many the product detected. And for most serious products, you'd find that the detection rates were between 99% and 100% - not a lot to choose between them. You could also do an "in the wild" test, against viruses that were actually out there, not merely in researchers' collections. And you'd expect that all serious products would score 100%.

Happy days!

It's totally different now - when I started looking at this recently, I couldn't believe how different. Testing is easy. I get sent, via email, dozens of trojans per week (the threat today is trojans, not viruses, although many people lump the two together as "malware").
So it's easy to assemble your zoo; they float in through your front door.

Testing products is easy too. There's VirusTotal, for example, which will show each of your samples to 50 or so products, and tell you which ones flag your specimen.

And the result is embarrassing. Or at least it should be embarrassing. I'm not embarrassed because I haven't had a product for 20 years. But AV vendors should be embarrassed, and the technical people there - well, I don't know how they can show their faces at conferences.

Because when I show a malware file that's just arrived by email to VirusTotal, it typically tells me that 90% of products fail to flag it. Sometimes 100% fail.

Here's one that just arrived a couple of hours ago. I say "one", actually, there were ten of them sent to me so far, each with a different from address.

Subject: CHAPS Remittance Advice (25/02/16)
   1.1 Shown   ~75 lines  Text (charset: ISO-8859-1)
   1.2   OK    ~61 lines  Text (charset: ISO-8859-1)
   2            47 KB     Application, "CHAPS_remittance_advice_75002891749.doc"

Please find attached your remittance advice.

If you do have any queries regarding this remittance advice, please contact:

Threadneedle (Supplier Reference beginning TP)

Tel No: 01330 069 014
Fax No: 02051 969 501
According to Metadefender and Jotti, Kaspersky flags it as a trojan downloader. Everyone else passes it as clean. According to VirusTotal, everyone (including Kaspersky) passes it as clean.

How can the people selling these products show their faces in public?

So what is protecting my systems?

1) I'm running linux, not Windows. 2) I've disabled the running of macros in my word processor. 3) I'm not stupid enough to load an email attachment unless I'm *sure* that I know who it's from, and it really was from that person.

For most people, 1) isn't an option, and they wouldn't know how to do 2. And I expect that 99.9% of people aren't stupid, but there are *so many* of these malwared emails, and some of them are *really* plausible, and I doubt if the criminals would be sending them out unless it worked at least sometimes.

I can see the problem, of course. The malware is emailed to me and many other people; the AV vendors only get to see it after it was mailed out. They could argue "Well, it's impossible to write a product that will flag these". But it isn't impossible to write a product that will protect users against many of the threats (see my previous blogs for how).

And if it is actually impossible to create a product that gives a useful amount of protection ... then say so, and don't sell snake oil.

No comments:

Post a Comment