A recent post on the How Software is Built blog (often a good read) seems to be very confused about open Vs closed source quality control processes, especially with respect to security vulnerabilities. Unfortunately this is one of those posts that reminds the reader that the blog is sponsored by Microsoft.
First we have the question:
In this day and age, isn’t it easy enough to quantify vulnerabilities?
A comment from “Swashbuckler” gets the answer exactly right. Quantifying known vulnerabilities is easy, but quantifying unkown vulnerabilties is impossible. No tool or QA process will allow you to quantify unkown vulnarabilities.
Software quality is not only about the bugs you know exist, it is also about the ones you don’t know exist.
OK, so that one may be an innocent mistake on the authors part, after all, it was phrased as a question (albeit a rhetorical one). However, the post then goes on to say:
I simply don’t get the same feeling from the open-source people we’ve talked to. When we’ve brought the subject up, the response is almost universally “many eyeballs,” and faith (without data) that “many eyeballs” is effective.
OK, I don’t know who the author has talked to, but it is certainly true that some projects, be they open or closed source, rely on haphazard QA processes. So this statement alone doesn’t bother me too much, but it is followed with:
Am I completely off base? Do things like the Linux kernel and Apache go through rigorous security reviews?
So the author has not talked to two of the communities where security is critical. If the author does not know the answer to this question then they have no right to make the previously quoted assertion.
The answer, by the way, is yes. Both Linux and the Apache web server go through rigourous security reviews.
Many people (errr… many eyes) use the cutting edge code in development and run their internal security tests before rolling out across their multi-million pound businesses or embedding the code in their multi-million pound revenue generating software/hardware products. The many eyes principle is notjust about people reading code contributions, although that is an important part of it. It is also about many eyes on the development as it happens, as opposed to after it is released as a QA’d product.
The author then asks:
Is there proof that “many eyeballs” in open source is at least as good as something like the Security Development Lifecycle in Microsoft?
Both Linux and Apahce, the examples chosen by the author see significant contributions from staff at companies like IBM, Sun, Oracle, RedHat, HP, Google, Amazon, Yahoo!, Facebook (and so on). Does the author really think that these multi-million pound businesses just take the latest Linux or Apache web server release without testnig it? Does the author really think that they don’t get involved with a new release during the QA stages of development? Does the author really think thay they sink developer resources into these projects but ignore the QA process?
Of course, there are also all the smaller businesses who work on development too, they are part of the QA process as well.
That’s what “many eyes” means. I can’t tell you if it is better or worse than any other QA process. However, I would bet that companies who use open source to drive their multi-million pound businesses feel that their own organisations QA processes are “at least as good as something like the the Security Development Lifecycle in Microsoft”.