A recent post on the How Software is Built blog (often a good read) seems to be very confused about open Vs closed source quality control processes, especially with respect to security vulnerabilities. Unfortunately this is one of those posts that reminds the reader that the blog is sponsored by Microsoft.
First we have the question:
In this day and age, isn’t it easy enough to quantify vulnerabilities?
A comment from “Swashbuckler” gets the answer exactly right. Quantifying known vulnerabilities is easy, but quantifying unkown vulnerabilties is impossible. No tool or QA process will allow you to quantify unkown vulnarabilities.
Software quality is not only about the bugs you know exist, it is also about the ones you don’t know exist.
OK, so that one may be an innocent mistake on the authors part, after all, it was phrased as a question (albeit a rhetorical one). However, the post then goes on to say:
I simply don’t get the same feeling from the open-source people we’ve talked to. When we’ve brought the subject up, the response is almost universally “many eyeballs,” and faith (without data) that “many eyeballs” is effective.
OK, I don’t know who the author has talked to, but it is certainly true that some projects, be they open or closed source, rely on haphazard QA processes. So this statement alone doesn’t bother me too much, but it is followed with:
Am I completely off base? Do things like the Linux kernel and Apache go through rigorous security reviews?
So the author has not talked to two of the communities where security is critical. If the author does not know the answer to this question then they have no right to make the previously quoted assertion.
The answer, by the way, is yes. Both Linux and the Apache web server go through rigourous security reviews.
Many people (errr… many eyes) use the cutting edge code in development and run their internal security tests before rolling out across their multi-million pound businesses or embedding the code in their multi-million pound revenue generating software/hardware products. The many eyes principle is notjust about people reading code contributions, although that is an important part of it. It is also about many eyes on the development as it happens, as opposed to after it is released as a QA’d product.
The author then asks:
Is there proof that “many eyeballs” in open source is at least as good as something like the Security Development Lifecycle in Microsoft?
Both Linux and Apahce, the examples chosen by the author see significant contributions from staff at companies like IBM, Sun, Oracle, RedHat, HP, Google, Amazon, Yahoo!, Facebook (and so on). Does the author really think that these multi-million pound businesses just take the latest Linux or Apache web server release without testnig it? Does the author really think that they don’t get involved with a new release during the QA stages of development? Does the author really think thay they sink developer resources into these projects but ignore the QA process?
Of course, there are also all the smaller businesses who work on development too, they are part of the QA process as well.
That’s what “many eyes” means. I can’t tell you if it is better or worse than any other QA process. However, I would bet that companies who use open source to drive their multi-million pound businesses feel that their own organisations QA processes are “at least as good as something like the the Security Development Lifecycle in Microsoft”.
The reason for asking “Am I completely off base? Do things like the Linux kernel and Apache go through rigorous security reviews?” is because we have interviewed a number of people, and the response so far has been “No, there isn’t a huge emphasis on secure coding and security testing. Many eyeballs. Many eyeballs.”
I agree, it’s hard to fathom that companies like IBM, Sun, Oracle, RedHat, etc, wouldn’t really focus on secure coding. It’s hard to believe that fortune 500 companies would be running an operating system that hasn’t gone through extensive security testing, but so far, I haven’t uncovered significant evidence to the contrary.
I have found no evidence that IBM, Intel, and RedHat train their people in threat modeling, for example. APIs like strcpy haven’t been banned from the kernel or apache development. If someone like IBM does write test suites to look at the security of sub-systems, I don’t see that those security specific tests have been checked in for others to run and improve on. There are tests for specific security features, but those are not the same as security focused tests across all the subsystems.
Again, maybe I just haven’t talked to the right people, and if so, I’d love to correct my stance, but based off a substantial number of interviews (you can look at the list of people interviewed so far at http://www.howsoftwareisbuilt.com), I’m left with the conclusion that Microsoft is leading the industry when it comes to building secure products. While I can’t comment on the number of unknown vulnerabilities, the data on known vulnerabilities seems to indicate that Microsoft products developed after the inception of their “Security Development Lifecycle” have fewer known vulnerabilities than the open-source counterparts.
Scott,
Thanks for taking the time to respond. Unfortunately, I don’t feel that your response provides the balance I’m looking for. I certainly agree that MS have significantly improved the level of security in their products in recent years. I commend them for doing so. However, I’m more than a little concerned that your unsupported claims of superiority are not helpful.
Please understand, I am not saying your claims are wrong, only that they are unsupported.
Specifically, you say:
“I have found no evidence that IBM, Intel, and RedHat train their people in threat modeling, for example.”
Have you actually looked for such evidence beyond the reviews on the howsoftwareisbuilt.com website? If so what depth of analysis have you made?
I decided to take your challenge and look at the interviews that mention security on the howsoftwareisbuilt site. My findings are summarised below. Note, I’m ignoring interviews with MS folk as I think we can all agree that MS have significantly improved their security QA processes in recent years. The question here is whether we can justify a claim that other companies do not have similarly robust security QA processes.
In the MySQL interview it is clearly stated that the many eyes principle is what makes things secure.
Stormy Peters is asked about the MySQL comment specifically, but does not directly address it in her answer.
Mark Gross (Intel) states that many eyes is important, but clarifies by saying that this means “You get people testing it.” Unfortunately, he does not expand on what he means by “testing it”, he could be referring to an ad-hoc process, he could be referring to a very dilligent internal process.
In the PostgresSQL interview Josh Berkus does not refer to the many eyes idea directly, but he observes that visible code is clean of “mystery functions that nobody understands”, thus fixes are easier to implement if a vulnability is found.
That’s it. The total number of projects specifically interviewed on this topic is one (MySQL), whilst three other people talk about it in some way. None of these companies are the ones Scott refers to in the previous comment or his original post.
Perhaps howsoftwareisbuilt.com would like to interview the companies and/or projects named. I’d be really interested to hear responses to the original blog post or to the following questions:
What do Red Hat do that enables them to claim that:
“Red Hat continually looks for potential security exposures and delivers tested security updates through Red Hat Network”
What do Oracle do that enables them to claim that:
“We have a complete Oracle and Linux security support, and offer support for Oracle Linux security tools”
What do IBM do that enables them to claim that:
“IBM will help you assess, detect, protect, correct and recover from security exposures in your IT and physical security environments in today’s on demand world.”
My point remains the same as in the original post. All three companies (and many others) bet their business, at least in part, on security. I cannot accept that they do so without having internal processes to manage the assocaited risks and I cannot accept that those processes do not feed back into the open source projects they use in their businesses.
Finally, Scott, you say “the data on known vulnerabilities seems to indicate that Microsoft products developed after the inception of their “Security Development Lifecycle” have fewer known vulnerabilities than the open-source counterparts.”. Can you please point us at this data. I am currently unaware of any reliable, unbiased and extensive data covering this topic area. As an unbiased, non-advocacy advisory service we would find such data invaluable.
An interesting article on E-commerce news written by PayPal’s Matthew Mengerink says:
“PayPal transacts more than US$1,500 every second of every day, with millions of people around the world relying on the robustness of its system.
It comes as a surprise to many people that PayPal runs such a large financial services company on an open source platform, but that’s precisely how we’re able to deal with the two competing demands our business model places on us: security and innovation.
Here’s some related commentary from Matt Asay.
The US National Security Agency uses a security enhanced version of Linux. I’d guess that they are pretty thorough about their security audits.