- Consumer Reports claims they installed the tested antivirus products, granted them Internet access, and then "spent weeks closely monitoring each product and noted how early, if at all, the manufacturer equipped it to detect newly discovered viruses." Exactly how did Consumer Reports know when these allegedly new viruses first appeared? And without knowing when they appeared, how could they possibly know how early a vendor provided detection? There is a way to measure response times scientifically, as is done by AV-Test.org, but such tests must be run in a carefully controlled environment and compile data from a much more significant timeframe than 'weeks'.
- Consumer Reports claims to have created 5,500 new virus variants (thanks guys, just what this world needs!) to test the products and further claims that their creations are "the kind you’d most likely encounter in real life." Let's hope not!
- Consumer Reports claims F-Secure AntiSpyware is the top-rated spyware scanner and that Lavasoft's Ad-Aware is fifth. Yet F-Secure's antispyware scanner is Lavasoft Ad-Aware, so how one can be first and the other fifth is beyond comprehension. (My own tests have shown that both tend to detect approximately 65% of the adware and spyware in-the-wild.) Interestingly, one of the only products able to remove 98% of active processes associated with the 100+ live adware and spyware infections used in my tests has been McAfee AntiSpyware 2006 - a product Consumer Reports places at second to last on their list. Another stellar - and free - antispyware has been the consistent (and otherwise consistently top-rated) Windows Defender, which Consumer Reports places last. And Sunbelt's CounterSpy, another consistently top-rated scanner, also received low marks from the Consumer Reports lab.
- Consumer Reports claims the tests were facilitated by Independent Security Evaluators (ISE) and that the president of that company, Avi Rubin, recused himself from involvement due to a conflict of interest. Perhaps that's so. Or perhaps Mr. Rubin was simply smart enough to know a bad idea when he saw one.
Admittedly, I may know very little about vacuum cleaners, cars, coffee pots, and many of the other things Consumer Reports tests - but I do know security software. The methods used, and the results construed from those methods, cause me to severely question the validity of any of their more mainstream reviews. I'm actually in the market for a new vacuum cleaner and a new coffee pot, and I'm sure of one thing - I won't be relying on Consumer Reports for buying advice.