CNET Review: More Questions Than Answers

CNET published a recent review of Norton AntiVirus 2006 and McAfee VirusScan 2006. The review measured performance and its accompanying graph claims 'longer bars mean better performance'. Based on that, CNET apparently thinks that the longer it takes to scan, or the longer it takes to boot a system, the better. In fact, the opposite is true - what you want are shorter scan times and shorter boot up times.

The CNET review also measures Frames Per Second using the popular Doom 3 game. According to the review, they achieved 21.8 FPS without the antivirus and 21.6 with. I'd argue first that FPS is a meaningless measure of antivirus performance; it's purely a measure of the graphics card capability. In any event, anything below 30 FPS is generally considered pretty much unplayable by most avid gamers, so one has to wonder at the rather abysmal 21.8 FPS benchmarked even without the antivirus.

But the real kicker in the review has to do with response times. CNET used times provided by Secunia for their virus response times. And while Secunia is a very reputable outfit, their times are based on press releases, not on virus definition updates. In short, the CNET review is simply a measure of how nimble the antivirus public relations departments are, not how quickly you or I received protection.

For example, looking at Sober.R, CNET erroneously claims the following times for detection:

McAfee 2005-10-06 at 03:47
Symantec 2005-10-06 at 5:20 hours
F-Secure 2005-10-06 at 11:11 hours

In reality, based on times the actual definition file updates were released and not when the press releases were sent, protection was made available as follows:

McAfee 2005-10-06 at 05:13
Symantec 2005-10-06 at 06:36
F-Secure 2005-10-06 06:45

But worse, listing only those three implies they were the first three responders. In fact, for this particular threat, McAfee, Symantec, and F-Secure all took 5+ hours to provide protection. Conversely, five vendors (Dr. Web, QuickHeal, eSafe, Nod32, and Fortinet) detected heuristically - i.e. without requiring specific updates. And of those detections that did require updates, several vendors responded far more quickly than Symantec, McAfee, or F-Secure, including:

BitDefender 2005-10-06 00:54
ClamAV 2005-10-06 01:00
AntiVir 2005-10-06 01:13
Kaspersky 2005-10-06 01:26 F-Prot 2005-10-06 01:50
Sophos 2005-10-06 03:07
Command 2005-10-06 03:42
Panda 2005-10-06 03:53
Kaspersky 2005-10-06 03:56

Independent response time testing is performed by AV-Test.org, a project of the University of Magdeburg and AV-Test GmbH, Andreas Marx. The test methodology is described in "Outbreak Response Times: Putting AV To The Test", published in Virus Bulletin magazine in February 2004.