Sometimes we are asked to give an opinion on a particular piece of open source software and its quality in comparison to a specific closed source alternative. Of course, with the sheer number of projects and products out there, it is often very hard to answer these kind of questions with any authority, and this means that we can often not give a detailed answer. On one occasion where I was personally asked this kind of question, I gave the usual disclaimer and set about asking what contacts I had in that specific problem domain what their opinion was (for my own edification as much as that of the questioner). One particular response I got back was interesting; I’ll paraphrase as the communication was not intended to be public. In essence the respondent – someone with long years’ experience in this particular area – told me that they had heard good things about the open source implementation but that in their opinion only an idiot would ever use it for ‘real world tasks’. It stood to reason, they argued, that open source must necessarily be buggier and less professional than closed source, and notwithstanding anything they heard to the contrary about the quality of this particular solution, they could not recommend anyone waste their time with it.
Now as I say, the OSS Watch staff are not experts in every software-intensive problem domain, and so we do not gainsay actual experts lightly. Even so, in this case I noted to myself that I might be seeing a certain amount of unsupported prejudice. The problem is that code quality is a notoriously hard property to assess. Even users of the same program can have radically different impressions of its quality, stability and efficacy. One approach to arriving at verifiable metrics of code quality is static program analysis, where software is used to analyse the source code of other software and identify where problems might occur. One company that offers static analysis software and services is Coverity, and over the last five years, partnership with the United States Department of Homeland Security, they have been periodically assessing the quality of selected large open source projects. As might be hoped, the picture has been one of gradually increasing code quality with each survey.
This year for the first time Coverity made a direct comparison of open source and proprietary code quality, and the results were interesting (you may need to register in order to receive the pdf of the report). In the open source projects they examined (Linux, PHP, and PostgreSQL) rates of software defects were lower than in the corpus of proprietary closed code with which they compared (0.45 vs 0.61 problems per 1,000 lines of code respectively). Of course, we must be cautious about such a circumscribed survey. The three projects they chose are well supported, mature and active. They also, in common with the proprietary comparators, use the Coverity software to identify errors as part of their development processes. Therefore one could conclude – and Coverity seem keen that we do – that the real lesson here is that using their software reduces error rates whatever your licensing or development model. Still, it is useful to have some more evidence in the discussion of open source vs proprietary code quality.
0 Responses to “Open source “matches proprietary code quality””