Author Archive for Rowan Wilson

Star pupils

Published by Rowan Wilson on January 27, 2012 in Community and Mobile technology. 0 Comments

Google has been in the news repeatedly over the last six months for closing down some of its many, many side projects. In general these are being mothballed in perpetuity, but in some cases, there is a transition plan. This is the case for Google Sky Map, and for our community it’s an interesting variation on the more traditional ‘open source it and hope it takes off’ approach that industry players like Nokia have tried in the past, and HP seem about to try again.

Sky Map is an application for Android mobile devices that was born out of the so-called 20% time that Google grants its engineers for the pursuing of personal projects. Back in 2009 when Sky Map was launched, one of the few hardware advantages that the few Android phone then on the market had over the iPhone was a hardware sensor compass. The ‘wow’ moment of the first public display of Google’s Android hardware (the G1) was a demonstration of how the Streetview service could be used in combination with the device’s compass to display information relevant to the direction you are facing. To build on this brief window of competitive advantage (the iPhone acquired a compass in its next hardware iteration) Google’s Pittsburgh office (then based on the campus at Carnegie Mellon University) developed and released Sky Map. The functionality – showing information about the night sky in the direction you were looking – was useful and educative and spawned tens of imitators over the next few years.

Given that the software no longer promoted a competitive advantage of Android hardware, and that the competition for apps of this kind has been getting tougher and tougher, it’s not entirely surprising that in Larry Page’s seemingly endless round of belt-tightening the project has been let go. Sky Map is returning to its roots on the Carnegie Mellon Campus:

“Today, we are delighted to announce that we are going to share Sky Map in a different way: we are donating Sky Map to the community. We are collaborating with Carnegie Mellon University in an exciting partnership that will see further development of Sky Map as a series of student projects. Sky Map’s development will now be driven by the students, with Google engineers remaining closely involved as advisors. Additionally, we have open-sourced the app so that other astronomy enthusiasts can take the code and augment it as they wish.”

This is an interesting approach. Although it’s not clear yet exactly what kind of student projects will be invited (computer science? astronomy? both? neither?) the idea of taking end-of-life, production code and open sourcing it to facilitate learning and teaching is a model I would like to see more generally adopted, for a few reasons.

Firstly, it is likely to teach open development methodologies to student software authors, something which is still notable by its absence in too many academic primary and secondary software development programmes. Secondly, it provides a compelling means of community engagement for the academic institution, opening a window into their teaching for the outside world and inviting collaboration. Thirdly, it advertises the skills of the students with a directness and accessibility that mere CV distribution cannot really match.

Although I’m not thrilled by Google’s year of the long knives (goodbye Google Sets *sniff*) solutions like the one proposed for Sky Map are genuinely exciting, and I’ll be watching for the resulting academic projects with interest.

Nginx and the Open Core model

Published by Rowan Wilson on January 6, 2012 in Community. 2 Comments

For years now the Apache HTTP Server has been by far the most widely used web server on the internet. Netcraft publish statistics on web server usage monthly, using a variety of metrics, and this month’s stats show an interesting change. While Apache HTTP Server is still miles in the lead, second place in the ‘active sites’ metric (meaning sites which are not just mothballed domain names) has transferred from closed source Microsoft web server IIS (Internet Information Server) to open source upstart Nginx (pronounced ‘Engine X’), released under the two-clause BSD license. Nginx has developed a reputation for speed and low resource requirements that has made it popular in a relatively short time.

So the fact that the top two slots in one of Netcraft’s surveys are now filled by open source web servers is interesting in itself, but there’s something more to this. Unlike Apache HTTP Server which is developed under the supervision of a US not-for-profit foundation, Nginx has recently become a commercial company offering paid support and successfully raising $3m in series A venture capital funding. As well as paid support, Nginx has announced that the intend to implement an Open Core model for their business going forward.

Now the Open Core model is what we used to call ‘proprietary extensions’, meaning that the open source code is supplemented with closed source paid add-ons for those that want them. In a way it is similar to the shareware model that did so much for PC gaming during the 1990s, bringing games like Doom to offices everywhere. One often cited problem with the Open Core model is, however, that users of the open source ‘core’ are at liberty to build competing open source versions of your proprietary extensions. Indeed you can find that ideological opponents of the partial freedom that Open Core embodies may be motivated to compete simply because of that ideological opposition, essentially enforcedly ‘opening’ the parts of the project functionality that you wish to keep closed. The only really effective defence against this risk is to be the best-resourced and most skilled team working on the code, thereby ensuring that your extensions cannot be easily replicated by competitors. So Open Core is an interesting strategy, in that it has drawbacks from both the purely ‘open’ point of view and the more traditional closed source approaches to software exploitation. In the past it has been accused of attempting to benefit from the ‘Halo Effect’ of open source while in fact leveraging closed methodologies for value realisation, but the fact that Nginx has managed to achieve so much in such a short period makes it a technology and a company to watch.

FOSS Focus

Published by Rowan Wilson on December 1, 2011 in Mobile technology. 1 Comment

I was thoroughly drawn in by the Amazon ‘Black Friday’ event last week, buying both a phone and a camera, against my better judgement and to the disgust of my bank manager. While trying to suppress my buyer’s remorse by searching the internet for all the marvellous capabilities of my soon-to-arrive devices, I noticed that the camera, a Canon Powershot SX220 HS, was one of the models capable of running a piece of open source software called CHDK released under the GNU GPLv2. This program leverages the fact that the camera will execute anything that look even remotely like a firmware update that is located on its SD card without requiring a digital signature, allowing an adjunct to the device’s firmware to be executed every time it starts up. You can even place the program on the SD card and select whether it is booted or not by changing the ‘write protect’ switch on the card.

Once the software is booted the user has access to an almost ridiculously long list of tweaks and features, including saving pictures as ‘RAW’ (meaning that the data from the camera’s sensor is saved to the card unaltered, rather than being crushed down into a smaller JPEG file), greater control over exposure times and the ability to construct more complicated ‘bracketing’, meaning that a series of shots can be taken with differing focal lengths or levels of white balance, allowing creation of HDR images and focus-stacked images. Even more geektastic is the ability to script the functionality of the camera using UBASIC or LUA, allowing a user to build functionality like time lapse photography and the taking of pictures only when motion is detected.

One question that remained with me, even as I contemplated spending more unwise pounds, was what Canon’s attitude was to this project. After all, some of the functionality that CHDK contains can be obtained from Canon in its more expensive models. They could close the technical loophole that allows the additional software to be run fairly easily, so one must assume that they do not see the existence of open source expansions of their equipment as a threat to their business model. Might they even see it as a selling point? Certainly it seems that running CHDK is likely to void your warranty, so perhaps the existence of a group of customers who opt out of expensive warranty provision is seen as a bonus.

Discovering this went some way towards alleviating the guilt of my spending. After all, I had got a large amount of functionality at essentially half price. But… if you want to run long scripts then you really need the attachment that lets the camera run off mains power. That’s not such a bargain. I could also probably do with a better tripod… Oh dear…

Carmack’s Reverse: a FOSS patent case study

Published by Rowan Wilson on November 18, 2011 in Business and Legal. 2 Comments

Just a quick one on the subject of open source and patents. John Carmack is well known in gaming circles as the lead programmer behind such classic PC and console games as Castle Wolfenstein, Doom (and sequels) and Quake (and most of its sequels). Carmack and his company id software are the originators of the ‘First Person Shooter’ genre of game which has in turn spawned such gigantic franchises as Call of Duty and Halo. As well as being technical pioneers, id has an interesting policy of releasing their old engine technology (the software which renders the game’s video and audio) as open source under the GNU GPL v2. This allows students of gaming software development to look at how real commercial games software is written, and also allows the games to be ported to new hardware platforms by volunteers. As the art and sound assets are not included with the code, this also generates a small market for licences to old id games – games which may well not run on more modern operating systems – in order to get the game data for use with the aforementioned ports.

Rage, the id game which uses version five of the id rendering engine (id tech 5, as it is known) has just been released. This is the point at which the source to the previous engine would usually be released as open source. However in this case there is a problem. Back when id tech 4 was being written for the game Doom 3 over the period 2000-2004, many games developers were looking into what were then cutting edge graphical technologies for inclusion in their games. One such technology was stencil shadowing, which accurately projects shadows from moving objects onto surrounding surfaces based upon the light sources which are illuminating them. This is hard work even for the specialist graphics hardware in PCs and consoles, and so any algorithmic optimisations that are possible are highly valued by the industry. So, various developers hit upon the same optimisation around the year 2000. This optimisation has come to be known as Carmack’s Reverse, even though it was first presented by William Bilodeau and Michael Songy of Creative Labs back in 1998. John Carmack discovered it independently some time later, and was perturbed to discover that Creative had already patented the process.

Faced with a choice between licensing the patent from Creative or making his code sub-optimal, Carmack decided to strike a deal with Creative that allowed him to use the technology at no cost. Perhaps coincidentally, id also agreed to use sound rendering technology by Creative in their game.

So the game was shipped and everyone was happy, until half a decade later when it came time to ship the code as open source. Obviously whatever deal was agreed between Creative and id did not involve making code available that embodied Creative’s patent under an open source licence for the whole world to use. In a tweet on the subject Carmack explained:

Lawyers are still skittish about the patent issue around “Carmack’s reverse”, so I am going to write some new code for the doom3 release.

It will be interesting to see how Carmack replaces the code, but the issue is also of interest because it illustrates the importance of keeping good records of the inbound IP in a software project. id’s lawyers caught what could have been an expensive potential infringement of Creative’s rights when they ’skittishly’ requested that Carmack rewrite the code.

So. Farewell Then Flash Player Mobile…

Published by Rowan Wilson on November 11, 2011 in Mobile technology. 0 Comments

(Apologies to E J Thribb)

Adobe’s announcement that it will be dropping Flash Player for mobile devices from its future plans has been widely interpreted as a victory for Apple, and in particular their late Chief Executive Steve Jobs. Perhaps because of his essay Thoughts on Flash, the absence of Flash technology from Apple mobile devices has seemed to be a personal decision of Jobs. In that essay Jobs made a group of points about exactly why he saw Flash as a detrimental technology, certainly for Apple mobile devices, and to some extent for Apple computers in general (“We also know first hand that Flash is the number one reason Macs crash.”) Competing phone and tablet makers pointed to their devices’ ability to run Flash, although not always that well. Apple’s refusal to engage with Flash on mobile led Adobe to declare that Apple mobile devices could not access the ‘full web’, particularly the video content that at that time was most frequently packaged as a Flash object.

In fact, Flash became so synonymous with web video packaging that it is easy to forget that it started life (as FutureSplash Animator back in the dark days of dial-up) as primarily a vector graphics animation package. It provided the possibility of small files containing large images and enhanced interactivity (this was before Javascript was well supported or unified). Gradually Javascript and broadband made these selling points more moot, and Flash then made the leap into bundling video codecs and providing a unified way for browsers to display video. Now, partly as a result of Apple’s stand against it, native video decoding in the browser combined with HTML 5’s <video> tag have once again made Flash less relevant. Now, it could be argued, the chief use of Flash is in rapid prototyping and dissemination of simple web games, many of which go on to spawn native titles on the bigger gaming platforms. Even that function is likely to be somewhat supplanted by HTML 5 apps in the future.

So has Flash as a whole been doomed by Apple’s tactics. It’s worth looking at that Steve Jobs essay again to find out. Among the many legitimate criticisms he levels (instability, insecurity, poor efficiency) Jobs also attacked the fact that Flash provided a path to developing mobile apps (both in the browser and compiled to native code) that was out of the control of the platform owner:

We know from painful experience that letting a third party layer of software come between the platform and the developer ultimately results in sub-standard apps and hinders the enhancement and progress of the platform. If developers grow dependent on third party development libraries and tools, they can only take advantage of platform enhancements if and when the third party chooses to adopt the new features. We cannot be at the mercy of a third party deciding if and when they will make our enhancements available to our developers.

This becomes even worse if the third party is supplying a cross platform development tool. The third party may not adopt enhancements from one platform unless they are available on all of their supported platforms. Hence developers only have access to the lowest common denominator set of features. Again, we cannot accept an outcome where developers are blocked from using our innovations and enhancements because they are not available on our competitor’s platforms.

Flash is a cross platform development tool. It is not Adobe’s goal to help developers write the best iPhone, iPod and iPad apps. It is their goal to help developers write cross platform apps. And Adobe has been painfully slow to adopt enhancements to Apple’s platforms. For example, although Mac OS X has been shipping for almost 10 years now, Adobe just adopted it fully (Cocoa) two weeks ago when they shipped CS5. Adobe was the last major third party developer to fully adopt Mac OS X.
Our motivation is simple – we want to provide the most advanced and innovative platform to our developers, and we want them to stand directly on the shoulders of this platform and create the best apps the world has ever seen. We want to continually enhance the platform so developers can create even more amazing, powerful, fun and useful applications. Everyone wins – we sell more devices because we have the best apps, developers reach a wider and wider audience and customer base, and users are continually delighted by the best and broadest selection of apps on any platform.

In other words: Apple did not want development tools for their mobile devices to exist which were not under their control. Jobs cited this as ‘the most important reason’ that he rejected Adobe’s technology. Yet only months after that essay Apple backtracked on the restriction and permitted Flash tools to compile Adobe Flash Actionscript applications for submission to the App Store. Indeed, if you don’t own a Mac, using Flash to generate iOS apps is one of the very few alternatives available to you. In fact, Flash Builder, the tool which does the actual compiling of Actionscript programs into iOS applications, is really the open source development environment Eclipse distributed with a proprietary Adobe plugin. Potentially the concession won by Adobe could lead to entirely open source tool chains for the development of iOS apps.

So while Jobs’ tauntings over the possibility of a robust and useful Flash Player Mobile (“We have routinely asked Adobe to show us Flash performing well on a mobile device, any mobile device, for a few years now. We have never seen it”) have proved prescient, in fact Adobe won a crucial concession from Apple almost a year ago. That concession widened the potential role of open source tools in developing iOS apps (it’s always been possible for Android). Adobe have also announced that they intend to ‘aggressively contribute’ to HTML5, perhaps indicating that they will be extending their development tools to allow Actionscript programs to be emitted as HTML5 web apps.

What really emerges from the struggles over mobile Flash is a strong sense of the entropy of the mobile device space at the moment. Rhetoric is deployed and attitudes struck, only for the originators to back away in a matter of months. App Store policies change monthly and with them the possibility of using open source code on the devices they serve. Competitive head-butting between closed source behemoths (like Adobe and Apple) can result in the opening up of data standards (as the pressure from iOS users has resulted in more HTML5 compliant video on the web – although let’s not get into the patents around those). For open source authors and proponents, the mobile space remains a changing and challenging environment. What the skirmishes around Flash demonstrate, though, is that the struggle of closed source vendors for competitive advantage can provide opportunities for free and open source code.

Build a better mousetrap

Published by Rowan Wilson on October 21, 2011 in Business, Development, Discussion, Mobile technology and Strategy and Policy. 0 Comments

“Build a better mousetrap, and the world will beat a path to your door” as Wikipedia informs me Ralph Waldo Emerson never quite said. The point – that real innovation sells itself – remains true today. Indeed it could be argued that the average consumer is more engaged with the heartbeat of technological innovation now than ever before, with software releases making headlines among the more traditional stories of war and celebrity.

Emerson’s non-quote does raise a question, however. How do we identify technology which is better? With mouse-traps there are some fairly obvious metrics relating to mouse mortality and cheese preservation, but not all inventions are as easy to benchmark. The last few weeks have seen anouncements of upgrades to the world’s two most commonly used smartphone operating systems: Apple’s iOS (version 5) and Google’s Android (version 4). Each brings a raft of new features, although in both cases it has to be said that these new features are no longer as core to the operation of the device as innovations in earlier versions. Voice-operated search and facial recognition are nice, but hardly essential elements of a mobile computer, at least for now. Perhaps lost in the combative comparisons deployed by proponents of each OS is the fact that a genuinely key ability – web browsing – is implemented on both platforms using essentially the same code: the Web Kit open source project. While newer functionality is added by Google and Apple to differentiate the competing products, it pays them to cooperate on key, unavoidable elements of their offerings. Given this, it’s fair to repeat the question – how do we identify real innovation? The newer differentiating features appear to be the cutting edge of endeavour, but their very newness is a demonstration that – up to now at least – they have not been essential elements of the technology in question. Some of them will die away despite their novelty, having never truly improved the invention that they embellish. Like a cheese grater on your mouse trap, it’s possibly a nice idea and undoubtedly novel, but how useful is it really? Only time will tell, and in the meantime better springs, and better browsers, are being developed.

So perhaps the question needs to be: “looking back at innovations that have proved to be key, how do they tend to develop?” Using the answer to this, we might be able to form some techniques for looking at our cutting-edge-but-possibly-pointless innovations and making guesses about their eventual utility. We might even be able to identify over-arching strategies for conducting and rewarding innovation…

Here we get into an argument that flared up earlier this month, when a video of Francis Gurry, the Director General of the UN’s World Intellectual Property Organization (WIPO) back in June was discovered by the internet commentating community. Gurry was speaking to sum up his views on a debate which had just taken place on ‘Accelerating Growth and Development’ in relation to invention and intellectual property. Gurry’s argument was seemingly  summed up by the headline on the BoingBoing article which drew it to the internet’s attention: “WIPO boss: the Web would have been better if it was patented and its users had to pay license fees”. Reading the article, though, even the quote that BoingBoing had pulled failed to use that emotive word ‘better’:

Intellectual property is a very flexible instrument. So, for example, had the world wide web been able to be patented, and I think that is a question in itself, perhaps the amount of investment that has gone into or would be able to go into basic science would be different. If you had found a very flexible licensing model, in which the burden for the innovation of the world wide web had been shared across the whole user community in a very fair and reasonable manner, with a modest contribution for everyone for this wonderful innovation, it would have enabled enormous investment in turn in further basic research. And that is the sort of flexibility that is built into the intellectual property system. It is not a rigid system.

Reaction to the video from proponents of open content and open source across the internet was voluble and aggravated. Gurry was accused of being ideologically indoctrinated and blinkered, tied to anachronistic models of IP registration and exploitation even in the face of the incredible growth and success of the web largely without the intervention of these models. In fact though, the most that Gurry says is that the web would have been ‘different’. Taken in the context of the statements which preceded it (and which you can hear by downloading the video), in which the value of the traditional IP systems had been questioned repeatedly, Gurry’s statements do not really support the distillation they were given, and which caused so much anger. He is trying to argue that the web could have grown within more traditional licensing structures. Whether he is right about this or not, he is not claiming here that it would have been ‘better’ under those circumstances.

The anger and confusion here are natural, though. The battle lines between proponents of the traditional and the more ‘open’ approaches to innovation (and here we should note that the buzz phrase ‘open innovation’ often itself refers to deeply traditional IP exploitation patterns) have long been drawn, and the forces on both sides are keen to tackle and destroy the arguments of their opponents wherever they see them. The web is often perceived  - with much justification – as a triumph of innovation outside the traditional IP exploitation framework. To hear someone perceived as being part of the old-guard even discussing it can seem presumptuous to some ears. Yet in reality the implied dichotomy here is simplistic. The open licensing movements themselves are underpinned by the arcane operations of traditional licensing and exploitation. While they may give these operations an innovative twist, they could not be enforced or defended without them. Conversely, Gurry’s example of why  the patent regime is beneficial fails to address the criticisms of openness proponents. He points to the publication framework implicit in the current patent system, and makes the comparison between the saxophone – which has fully documented design documents available thanks to its having been patented – and the violin – where many secrets of producing the greatest instruments have been lost through secrecy and the passage of time. This critique – while interesting – is almost wholly inappropriate as a defence of the current system in opposition to more open models. In the modern case, both models involve complete publication – the distinction lies in how benefits are reaped from exploitation and by whom.

Given the frequent failures of either side in this debate to engage with what the other is actually saying – illustrated by this sad tale –  it’s not surprising that telling which innovations are better remains hard. While ideology is important, it can often obscure our view of what actually matters to most people: how many mice are killed (or indeed captured).

SaaS – Who shares wins?

Published by Rowan Wilson on September 28, 2011 in Legal and Software. 0 Comments

Fairly often we are asked by projects we advise whether they can make an open source release of their code while controlling whether others can make money from it. In fact, we get asked it so often that it now has a place in our FAQ document:

3.8. Can I restrain commercial reuse of my code using an open source licence?

Not directly, no. Point 6 of the Open Source Definition states that a pre-requisite for any licence to be considered open source is that it should make ‘No Discrimination Against Fields of Endeavor’. Commercial exploitation is a field of endeavour.

Having said this, some open source licences make code that they cover less appealing for commercial entities to include in their products. In general, licences with some element of ‘copyleft’ such as the GNU GPLv2 or the Mozilla Public License will compel commercial reusers to make some or all of their own code open source. Permissive licences such as the BSD License and the Apache License v2 on the other hand allow the code they cover to be added to a closed source project without compelling any other code to be open source.

Recently we received a novel variation on this query, and I have adapted my answer to be a post for this blog. It’s not yet a frequently asked question, but it may well become so. The questioner had written some code which implements a network service, and wanted to know if they could make an open source release while simultaneously discouraging others from selling hosted services based upon the code…

There are free and open source software (FOSS) licences which make conditions about this kind of use of the software that they cover. The Open Software License v3 and the GNU Affero GPL v3 are examples. There is a brief explanation of this kind of licence condition on the OSS Watch site.

All that these licences require, however, is that licensees who deliver functionality of the software over the network must make the source code to their version of the code available to users. This ensures that improvements to the software are released back to the community, but it may not act as a strong disincentive to those providing the service; after all, it does not compromise their core offering of network services. Contrast this with the more traditional model of distributing the software itself, where basing a product on (for example) GPL v2 code would mean that your own product must also be GPL v2 licensed, compromising the ability to charge for licences;  after all, customers could onwardly distribute the product to other potential customers, undermining the business model.

The second – more traditional – example there is the basis for the common dual licensing FOSS business model. The project provides a ’community’ version under a copyleft FOSS licence and a commercial licence for businesses wanting to build a product on the code base for which they charge a licence fee. The model works because the copyleft licence has conditions that the business will pay to avoid. In the ‘application service provider’ model we are discussing here however, the condition that source code must be provided may well not be onerous enough for the application service provider to pay to avoid, particularly if they have not invested in improving the code in any way.

The upshot of this is that – if one wants to restrain competing businesses from using the software to provide a network service – a FOSS licence may not be the optimal choice. Having said that, it might be worth examining the motivation behind restraining competitive network service provision in the first place. In the worst case, where the competitor uses the code unmodified and therefore sacrifices nothing by making that source available freely, they are also necessarily behind the project maintainers in terms of consideration of improvements and new features. Given a choice between buying hosting from the originators of the software and a copy-cat business which makes no contribution to the code base, customers may well choose the former. Conversely, if the competing business is actively improving the code and making those improvements available as a result of the conditions of the licence for the initial project, this could be seen as a positive outcome that benefits the entire ecosystem and leaves businesses to compete on quality of service rather than code quality.

Standard alchemy

Published by Rowan Wilson on August 12, 2011 in Community. 0 Comments

One of the questions that comes up over and over again when we talk to software authors about licence choice is: “what is each type of licence especially good for?” Personally, when discussing permissive licences I tend to point to the imperative of pushing an open data standard as a prototypical use case. Luckily for me, computer graphics industry giants (and competitors) Lucasfilm and Sony Pictures Imageworks have just released v1.0 of their 3d scene format Alembic to prove the point.

Anyone who has dabbled in 3d computer graphics, perhaps using the FOSS modeller and renderer Blender, will have noticed just how many competing object and scene description formats there are out there. While this is a minor inconvenience for amateurs like me, it becomes a massive and expensive problem for visual effects production houses and their clients. Complex animated objects and scenes created in one application have to be laboriously exported frame by frame and reconstructed in the target application, often requiring duplicated effort to complete the import. Alembic avoids this by allowing the compression and exportation and importation of entire complex animated sequences.

So for CG professionals there is a clear need for something like Alembic (the name is taken from the piece of chemistry glassware which distills products and delivers them to another container). However for the creators of the individual software packages the idea of a simple interchange format is not a high priority; after all, why should they make it easier for clients to use another company’s software? It’s natural, therefore, that it is the clients themselves who have produced the standard and software which ’scratches this itch’. Indeed there’s a precedent in the same industry. Back in 2003 Industrial Light and Magic release OpenEXR, a data standard and implementing software that allowed the interchange of so-called ‘HDR’ images (images with a greater range of luminance than standard image formats can store). In eight years OpenEXR has become a widely-used standard. With any luck, Alembic will follow in its footsteps.

“Recipe for Rip-Offs”

Published by Rowan Wilson on August 2, 2011 in Community, Procurement, Standards and Strategy and Policy. 0 Comments

Here in the UK the Public Administration Select Committee has been looking into the poor record government has in procuring IT systems. The title of their report “Government and IT- “A Recipe For Rip-Offs”: Time For A New Approach“ serves as a neat summary of the content. Stating the problem, the report says

The UK has been described as “a world leader in ineffective IT schemes for government“. There have been a number of high cost IT initiatives which have run late, under-performed or failed over the last 20 years including: the Child Support Agency’s IT system, the IT system that would have underpinned the National ID Card scheme, the Defence Information Infrastructure Programme, the implementation of the Single Payments Scheme by the Rural Payments Agency, and the National Offender Management System (C-Nomis).

The main problem, the report says, is that the Government does not have the internal skills to specify and procure IT systems. As a result they tend to rely on large external contractors to manage the process of developing IT systems (and to subcontract to smaller businesses where necessary) . Naturally this involves handing over very large amounts of both cash and power to the ‘head’ contractors, and it is this complete externalisation of the ‘IT customer’ function that the report points to as the key failing in previous large government IT procurements. The answer, therefore, is to get better IT management skills within departments and take on the management of the smaller subcontractors themselves.

This is not the only failing identified. It seems that Government also tends to ‘gold-plate’ (over-specify) security requirements even on systems that do not require it. The report also criticises the tendency to see IT projects as a distinct kind of problem rather than an exercise in change management like any other. Nevertheless, it is the ‘externalisation’ problem which looms largest in the report’s somewhat gloomy findings, and it is in this context that the issue of open source arises.

Early on the report identifies the creation of ‘a level playing field for open source software’ as one of the approaches to solving the problem of Government IT that had already been suggested. In the recommendations, we find that open source is mentioned in the context of providing an open data platform for Government-held data which could be developed upon by third parties to provide analysis and manipulation applications. While both of these suggestions are sound in themselves, I think it is in the core recommendation that we can see the best opportunity to realise value for the UK taxpayer from open source software and development.

While there are very large scale corporations offering open source solutions, the majority of bidders for Government IT contracts offer closed source solutions, often with the bidder themselves retaining ownership of the IPR in the resultant code and licensing it under very restrictive terms. If the current reforms succeed in getting departments to break down IT procurements into smaller interoperating sections and invite bids for these from smaller, more agile developers, the opportunity for existing successful open source projects to be the bases for Government IT solutions expands. Assuming that the newly-acquired IT experts within departments are able to meaningfully engage with the communities around these projects – both through their hired developers and as users themselves – then huge amounts of value in terms of code, user requirements and expertise which are currently locked into closed, non-functioning projects will be available for the good of the community at large. The projects themselves will learn how to interact with Government clients, and software components of general application will find their way back into the public space to benefit other large-scale users.

All of these benefits, though, depend both on an openness to the use of open source software but also on expertise in managing the relationship with that software’s community. So while I welcome heartily the proposal that Government acquire the IT skills to take a hands-on role in managing their IT procurements, I hope that those IT skills will include expertise in exploiting the unique benefits of joining an open source community.

The PSN hack and open source

Published by Rowan Wilson on May 6, 2011 in Community. 0 Comments

I’m one of the people who has recently (perhaps in an excess of caution) cancelled their credit card because of the security breach of the Sony Playstation Network. Now you might wonder what this has to do with open source, but bear with me. Back in 2004 I went to a conference in The Hague about open source in the secondary software sector (meaning industrial sectors where software was a part of their product but not the core offering). One of the companies there was Sony Computer Entertainment. The presenter explained that Sony was a very open source-friendly company, and that within the development division in Japan Linux desktops were the norm. The presenter also pointed to the Linux installation kit that Sony had released for their then-current games console the Playstation 2 (PS2), and advised us to look out for more Linux-related tie-ins in future games consoles. True to their word, two years later the Playstation 3 launched with the facility to install Linux in the basic model. True, you could not access most of the console’s advanced hardware via this ‘Other OS’ option, but it was a nice gesture, and generally appreciated by the open source community.

Unfortunately, three years later, when Sony released the revised ’slim’ version of the console, they decided to remove the option to install Linux. Sony said that the change was necessary as they wished to focus on the gaming side of their console, and could no longer support the work necessary to create and test Linux drivers for the new hardware revision. Cynical observers commented that the removal was more likely due to the progress that hobbyist developers had made in unlocking access to the PS3’s more advanced hardware features via the ‘Other OS’ feature, thereby threatening the monopoly over the approval of advanced software that Sony then had. Indeed, shortly after the release of the PS3 slim, hobbyists succeeded in circumventing the PS3’s security features and progressively gained full control of the machine.

This brings us to the first open source-related lesson of this story – withdrawing freedom is hard. As the technical team who finally overcame the PS3’s security pointed out the removal of the ability to install Linux drew their attention to the console. Having sold the console as – in part at least – a venue for open development and hobbyist coding, Sony’s removal of this capability struck some as unjust, and motivated them to forcibly right what they saw as a wrong. Now I am not trying to equate the open source community with technologists who overcome third-party control mechanisms like the protection of the PS3; while there is some cross-over, the activities are essentially distinct. It can be said, though, that Sony used freedom to develop and open-source friendliness as marketing tools (particularly effective in a market where their chief gaming rival was perceived open source opponent Microsoft) and their decision to withdraw that freedom was (a) highly resented and (b) in the end ineffectual.

Moving to Sony’s more recent problems, their decision to take robust legal action against one of the technologists responsible for overcoming their console security angered many, and led to so-called DOS or Denial of Service attacks against the Playstation Network (PSN) – the service which enables online gaming and sales of downloadable content for Sony consoles. In the wake of these DOS attacks, Sony deactivated the PSN for a long period of maintenance. Many surmised that this was intended to allow changes to the system that mitigated the loss of control of the console platform itself that Sony had experienced. In fact, as the first link above shows, Sony’s PSN had been hacked and the personal details of approximately 100 million users such as myself had been accessed by unknown third parties. At the time of writing Sony has not given details of how the hack was effected, but rumours on the internet – many pre-dating the closure by several months – claimed that Sony may have been using outdated versions of popular open source software on their PSN servers. Professor Eugene Spafford of Purdue University gave some of the details of these rumours when testifying yesterday to the US House Energy and Commerce Subcommittee on Commerce, Manufacturing, and Trade. Professor Spafford mentions outdated versions of the Apache httpd server as a potential attack vector in the hack, basing his remarks seemingly on unspecified postings to security-related mailing lists. While I have not been able to locate the postings he is talking about, it is certainly true that purported logs of the efnet IRC channel #ps3dev have been circulating that contain claims very similar to those Professor Spafford raises.

This brings us to our second open source-related lesson from this ugly situation – even the best- maintained open source is only as secure as its last release. Clearly we have no way of knowing what happened in the PSN hack, but no-one should rely on the security of open source – or indeed any software – who does not also commit to keeping their installations up to date.

PS. Professor Spafford’s prepared remarks for the subcommittee (pdf)) are also interesting in that they recommend a system of consumer data protection that closely mirrors the UK’s own Data Protection Act.