October 2, 2009
With so many scareware purveyors about, it can be difficult for consumers to tell which antivirus product is legitimate, much less which of the legitimate antivirus products are best suited for their own needs. The problem can be exacerbated if you don't understand who tests what and how to interpret what the scores mean.
One recent example, performed by NSS Labs, was a test focusing on detection of socially engineered malware. Those test results give a glowing review to Trend Micro Internet Security Suite. Unfortunately, the majority of the praise may be unfounded. Read the test report closely and it's pretty clear that while Trend Micro Internet Security Suite performed well on blacklisting and whitelisting of the 3,243 known URLs that were used in the tests, when confronted with an actual executable Trend Micro detected and blocked only 5.5% of the malware. The NSS Labs test also specifically excluded any exploit detection, a critical omission given that vulnerability exploits to deliver malware have never been higher.
Another example can be seen with the recent (non-beta) entry of Microsoft Security Essentials. While MSE stacks up well compared to other free antivirus (third highest according to tests published in PC World and provided by AV-Test.org), free antivirus lacks many of the proactive features required to defend against today's malware. The for-pay antivirus products, in particular the security suites, combine reputation, behavior analysis, and often even sandboxing along with signature-based scanning. The free antivirus typically consists of signature-based scanning only and thus simply cannot afford the same level of protection as can the paid-for protection. Not to mention that with 30,000+ new malware samples discovered each day, even the best signature scanners are struggling to keep up.
Virus Bulletin's VB100 award is another test that is often subject to controversy and of which results are typically misconstrued. The VB100 is actually one of the oldest running antivirus detection tests and relies heavily on something known as the WildList. The WildList is a collection of malware that has been vetted by at least two distinct reporters. (A discussion of the usefulness of the WildList can be found at: Why Getting Rid of the WildList is a Bad Idea.)
The VB100 tests set a minimum bar - if a product is routinely submitted to Virus Bulletin for testing, you can be assured that the scanner is legitimate, reputable, and making every effort to ensure high quality and capability of their scan engine. Trend Micro is one of the rare vendors who refuses to participate in the VB100 testing, having announced a 'boycott' after alleged consecutive failures on the tests.
The VB100 tests also include checking for false positives, an anomalous detection that can be highly disruptive and sometimes even costly to users. One false detection of a legitimate file, and the scanner fails this portion of the VB100 testing. Scanners that routinely submit for VB100 testing try hard to ensure false positives don't occur.
Perhaps most importantly, the VB100 tests measure both on-demand (pure signature) and on-access (real-time) protection. Real-time protection is where proactive features such as behavior analysis kick in. The on-access tests are actually the closest proximation of the real user experience and thus play an important role in determining the suitability of an antivirus scanner. In other words, while the number of malware tested may seem limited given the > 3 million malware around today, the nature of the tests are rigorous and do provide a very useful means to separate the wheat from the chaff. Something not every test facility is able to do.
