A couple of decades ago, I got really fed up with the quality of antivirus product testing. So I invented "The Perfect Antivirus". My thought was, if you've devised a product test protocol, then you should apply it to "The Perfect Antivirus", and if the product test agrees that this is a good product, then the product test is rubbish. Here's how it works.
The Perfect Antivirus
I shall now give you, free of charge, an antivirus that if used correctly, detects all past, present and future viruses, never gives a false alarm, and has a zero cost. Skeptical? Then watch carefully ...
Echo %1 is infected by a virus!!!
You'll agree, I think, that P1.BAT will detect all past present and future viruses. That alone meets the "mathematically impossible" task! But, I hear you thinking, aren't there rather a lot of false alarms? Well, you didn't say you wanted a low false alarm rate....
OK, OK. I'm used to projects where the user specification changes in the middle. Never mind. I can deal with the false alarms ...
Echo %1 is NOT infected by a virus!!!
You'll agree, I think, that P2.BAT will never, ever, tell you that you have a virus when you don't. Of course, it has a pretty poor detection rate. I admit that. But I can fix it. See here ...
Echo Is %1 a virus? (Y/N)
If the user types "Y", you run P1.BAT. If the user types"N", you run P2.BAT. Remember what I promised you? An antivirus that if used correctly, detects all past, present and future viruses, never gives a false alarm, and has a zero cost.
So what can we learn from this?
1. If something is superb at detecting viruses, it's no use if it gives a lot of false alarms.
2. Anything that relies on the user to make a correct decision, on matters that he is not likely to be able to decide about, is useless.
3. You can receive something that is exactly what the salesman promised to deliver, and it's nevertheless useless.
4. If an antivirus product test would give a 100% score to "The Perfect Antivirus" then the test is useless.