Threat Reports: You Don't Know What You Don't Know

Saturday February 3, 2007

You don't know what you don't know. I was reminded of that recently when I received an email that started off with "to the best of my knowledge" and which went on to reveal a faulty conclusion based on incomplete data. I was reminded of it again today as I was comparing data from various antivirus vendor threat reports.

There was nothing wrong with the reports, mind you. They are prepared with careful attention to detail, data is culled, massaged, and presented in - I believe - a forthright and truthful manner. The charts are colorful and the writing is strong and compelling. The problem is the data itself.

Signature-based detection
For starters, any time you're dealing with signature-based detection, threats can only be identified after they are, well, known. Which in some cases means the report itself is more a report of the signatures available to a particular vendor than the threat landscape itself. Of course, today's vendors do have a tremendous knowledgebase of malware and its associated signatures. But with targeted attacks increasing and the potential for new threats to sneak under the radar at least in the short term, one is sometimes left wondering more about what isn't being seen, rather than what is. Or as the Sophos 2007 threat report puts it,

"Sophos has seen malicious code on websites being altered on average seven times a day. Some of the more commonly encountered adware is also changed and repackaged frequently by its authors in order to evade detection by security products."

Not a wide enough lens
The second problem with threat reports is that they are focused only on the areas in which that particular vendor has insight. As an example, ScanSafe provides filtering for HTTP web content and IM. ScanSafe's Annual Global Threat Report notes they detected 435 new viruses in 2006. Yet the Sophos threat report for that same period notes that Sophos discovered nearly a hundred times that number of new threats, or 41,536. Does that mean ScanSafe failed to detect 41,101 new threats? No. The disparity is a result of one company having a larger focus area than the other.

Timing is crucial
MessageLabs, deservedly considered the expert in all threats traversing by email, has a creation date on their 2006 threat report of December 18, 2006. This means, given time for writing and publishing, the data collection for that report likely stopped sometime in October. In this rush to market, the report gives only a single mention of the Stration worm - an email worm that spawned 1000 new variants in November 2006.

Conclusion: These constraints don't make the reports any less meaningful, nor should they even be considered in a negative sense. Each of the reports provides interesting insight based on that company's specific experties with that specific corner of cyberspace and for that specific period of time. It's just that they don't know what they don't know. And that's why you should read all of them.