Firefox, IE, Opera and Safary all equally safe?

Thursday, September 13th, 2007

NU.nl reports about a (English) Report about the safety of the web. In it, CA reports:

Browsers are one of the most commonly used applications today. Many people believe that Mozilla Firefox is more secure than Microsoft Internet Explorer, but their vulnerabilities are on par. In the first half of 2007, NIST reported 52 vulnerabilities in Internet Explorer of which half were medium or high severity. And there were 53 vulnerabilities reported in Firefox of which almost half were medium or high severity.

The numbers are climbing. In 2006, 96 vulnerabilities were reported in Internet Explorer and 103 reported in Firefox.

Even less popular browsers have more security holes. More than double the vulnerabilities have been reported in the Opera browser. NIST reports 14 vulnerabilities this year versus seven last year, and more than half of this year vulnerabilities are medium or high severity.

Apple Safari has 19 newly reported vulnerabilities this year nearly twice the number reported last year, and half of them are medium or high severity.

When will researchers understand that the number of vulnerabilities reported / fixed are not a good way to determine how secure an application is? The problem is either that these people don’t understand software development, or that these people wish to backup their pre-determined claims with hard evidence, so they start looking at reported vulnerabilities. It doesn’t work that way, unfortunately. There are way too many variables not accounted for:

• First off, where are the sources for their data? They only mention NIST, but no criteria they looked at at all. This immediately invalidates their findings.
• They look at ‘reported’ vulnerabilities, but these are third-party reports. Does NIST only reports vulnerabilities listed in the application’s release notes?
• Do these statistics include reported, but unfixed bugs? Firefox maintains an open bug reporting facility where every user can report bugs. Not all of those vulnerabilities may have been fixed. Are those included in the statistics? If so, how can they compare those reports against the reports done on a closed bugtracking system such as IE, Safari and Opera?
• Where is the proper trend analyses? “In 2006, 96 vulnerabilities were reported”. Trend analyses should be done over multiple years, IMHO.
• What were the severities of the reported/fixed vulnerabilities? For all we know, IE had 100 minor problems that were only exploitable when the moon was full and it was friday the 13th, but Firefox had 50 extremely severe vulnerabilities.
• What are the sources for the severities? The vendor? Hardly reliable information, as vendors like to downplay their own vulnerability’s severity.
• The report doesn’t take in account the user-base of the products. Nobody uses Firefox, Safari and Opera, whereas everybody uses Internet Explorer. That makes it a much bigger target for black-hat exploiters. No, it doesn’t say anything about the security of a product from a technical point of view, but it does from a practical point of view.
• The report mentions, briefly, the security of third-party browser plugins such as Flash, Java, etc. But they make no relation to the different browsers. Firefox has a very easy to install and use Flash / Java blocker. It also has a very good Javascript blocker. Javascript is probably the number one source of vulnerabilities in Firefox. Since it’s not installed by default, I can understand they don’t focus on this, but at least the security potential for Firefox is higher because of this.
• No mention is made about ActiveX. ActiveX only works on Internet Explorer and is a HUGE contributor to security vulnerabilities.
• One of the most important things: Reality! People like to ignore it (which I can understand), but that doesn’t mean it doesn’t exist. How many vulnerabilities have actually led to exploits in the wild?

In defence of CA, their report doesn’t specifically say that Internet Explorer, Safari, Firefox or whatever is more secure than the other. They just imply it. As usual, media outlets are twisting the view on reports in order to make for better news and scare-mongering.