OSS Watch has been participating in the development of Apache Rave, a ‘next-generation portal engine, supporting (Open)Social Gadgets as well as WC3 widgets’. As Sander observes in this blog, the Rave ecosystem is made up of a ‘diverse range of collaborators’ from both the academic and commercial sectors. These partners are sharing resources in order to build a critical piece of software at lower cost as well as to increase innovation around that product.

A few days ago I posted an evaluation of the Apache OpenOffice project’s journey through the Apache Incubator (all code entering the Apache Software Foundation (ASF) must pass through the incubator). That post looked at what makes an Apache project different from many other open source project. This post repeats many of the same points, but rather than examine them from the point of view of OpenOffice I will examine why predominantly academic team behind Apache Rave chose to go to the ASF.

In Apache projects, a Project Management Committee (PMC) oversees each project on behalf of its users, contributors, committers and the foundation itself. Upon entering incubation the PMC is guided by mentors from the foundation. Upon graduation mentors either retire or become equal members of the PMC. For the Rave community the provision of mentors meant that the project team could avoid the mistakes of many other open source projects. As a result, the team got an honourable mention in the Black Duck Open Source Rookie of the Year awards. Not bad for a team with no significant knowledge of open source software development on a large scale. Now it has graduated, it no longer has mentors actively overseeing its work, but it still has the backing of over 100 full Apache projects and another 50-odd incubating projects.

New committers and PMC members are elected by the PMC based on merit. It should be relatively easy for anyone to gain influence on an Apache project. In the ASF this is achieved through rewarding merit. If you contribute to the project you are rewarded with influence over the project. In environments where staff turnover can be high, such as academic research, this is important with respect to continuity. It also removes the opportunity for someone to insist on a level of control based purely on the cash they wield. In an Apache project it is all about the delivery.

All decisions unrelated to individuals happen on the public mailing list, discussions on the private list is kept to a minimum. This behaviour has no special bearing on academic projects compared to non-academic projects. For both types this rule ensures maximum inclusivity which results in maximum engagement with potential contributors.

‘If it didn’t happen on the dev list, it didn’t happen’ – meaning no decision about the project can be made outside of the public development list. Proposals can be drawn up elsewhere, but decisions occur on the public list. Academic projects, like open source projects in general, often involve collaborators from a variety of geographic regions. This can make it difficult to ensure that everyone is kept informed and engaged. Apache projects require that all significant decisions are made in public so that no participant (or potential participant) is excluded from the process.

Where possible, decisions are made by consensus reached through discussion. There are voting rules but the ASF prefers not to have to vote. Apache Rave began life as a merger between three pre-existing projects. It was important that all three parties were equally engaged in the project. Had there been a pre-defined leader this would, probably, have made some participants feel less engaged. Initially the consensus driven approach can be hard to understand, however, over time natural leaders emerge in specific areas of the project. At this point consensus is easily achieved since each decision is led by the person best equipped to lead it.

Releases are created according to the ASF’s licence requirements. The Apache License is a permissive licence that allows anyone to do anything they want with the code. This allows for maximum flexibility in business cases for engaging with the project which in turn encourages third party contributions. Whilst conforming with Apache policies is more onerous than might be found elsewhere, they are designed to ensure that people can use and contribute to your software with minimal legal risk. Risk is something that universities and companies alike tend to avoid.

Trademarks and logos used by ASF projects belong to the ASF. Protecting trademarks is an important part of open source software. By running the Rave project inside the ASF much of the legal infrastructure and experience is in place should an issue arise in the future.

Apache projects are managed by a diverse group of people, each representing their own interests within the project. Apache decision making processes prevent ‘block votes’ controlling the process by ensuring each voice is equally loud. A number of people are contributing to Apache Rave, each with their own motivations. Each contributor must be assured that what they do today will still be useful tomorrow. Apache projects adopt a model that means it is not possible for third parties to gain control of a project. Consequently, researchers and product developers do not run the risk of losing influence over the code.

As can be seen from the above list of required behaviours found in Apache projects, the focus is on ensuring the project provides maximum opportunities for collaboration and innovation. There are other ways of achieving this but for the initial participants in Apache Rave (Universities of Bolton, Oxford and Indiana, SurfNet, Mitre Corp. and Hippo) the ‘Apache Way’ was deemed to be the most suitable. The same can be said of Apache Wookie which is used in Rave and was also helped by OSS Watch as it moved to Apache.

If your project wants to explore the opportunities that foundations (not just the ASF) can offer your project OSS Watch is here to help.

Now as I say, the OSS Watch staff are not experts in every software-intensive problem domain, and so we do not gainsay actual experts lightly. Even so, in this case I noted to myself that I might be seeing a certain amount of unsupported prejudice. The problem is that code quality is a notoriously hard property to assess. Even users of the same program can have radically different impressions of its quality, stability and efficacy. One approach to arriving at verifiable metrics of code quality is static program analysis, where software is used to analyse the source code of other software and identify where problems might occur. One company that offers static analysis software and services is Coverity, and over the last five years, partnership with the United States Department of Homeland Security, they have been periodically assessing the quality of selected large open source projects. As might be hoped, the picture has been one of gradually increasing code quality with each survey.

This year for the first time Coverity made a direct comparison of open source and proprietary code quality, and the results were interesting (you may need to register in order to receive the pdf of the report).  In the open source projects they examined (Linux, PHP, and PostgreSQL) rates of software defects were lower than in the corpus of proprietary closed code with which they compared (0.45 vs 0.61 problems per 1,000 lines of code respectively). Of course, we must be cautious about such a circumscribed survey. The three projects they chose are well supported, mature and active. They also, in common with the proprietary comparators, use the Coverity software to identify errors as part of their development processes. Therefore one could conclude – and Coverity seem keen that we do – that the real lesson here is that using their software reduces error rates whatever your licensing or development model. Still, it is useful to have some more evidence in the discussion of open source vs proprietary code quality.

At the same time, cloud technologies deliver better and better IT services. From email and content storage to complex computing and development platforms, users can access clouds via simple browsers, thus eliminating the need for end-user applications and high-power computers.

In UK Higher Education, cloud solutions are an integral part of a JISC programme aimed at helping universities and colleges deliver better efficiency and value for money through the development of shared services. As pointed out by Rachel Bruce, JISC’s Innovation Director for digital infrastructure, cloud solutions are increasingly attractive to HE institutions. They allow universities to reduce environmental and financial costs, share the load of maintaining a physical infrastructure, be flexible and operate on a pay-as-you-go basis, access data and applications from any location, and make scientific experiments easier to reproduce.

Mobile and cloud technologies are here to stay, and open source already plays a key part in the new market emerging from their cross-pollination, as the Future of Open Source survey indicates.

What are the effects of these two giants moving towards each other? What are the technological and economic implications of their intersection?

Open Source Junction 3 is an event specially designed to help you answer such questions by bringing together scientific innovation and business entrepreneurship to showcase early success and facilitate new collaboration in this sector. Over two days delegates from industry and academia will meet at Trinity College in Oxford to hear about highly innovative mobile cloud projects and work together to build industry-academia partnerships in this field.

One of the featured projects at OSJ3 will be ‘Access King’s Global Desktop’, which was implemented to support a new mobile working strategy at King’s College. Alex Hove from Getronix will describe King’s successful migration to a private cloud platform delivered over the JANET network, which provided some 25,000 staff and students with access to core applications from almost any device and any location.

Ajit Jaokar, founder of the publishing and research companyfuturetext, will highlight the technical architecture challenges facing a world of 50 billion connected devices predicted for 2020. He will discuss how the next generation web and new networking technologies such as white space networks may be able to address these challenges, in the context of the European ‘Smart cities’ initiative.

The ASTRA research project develops and tests low cost platforms capable of delivering scientific instruments into the stratosphere. Steve Johnston, Senior Research Fellow at Southampton University, will describe the hardware, data communications and software applications required for stratospheric flights, including the recent success of using Windows Phone 7 dataloggers with GSM communication and the Azure cloud for back-end processing.

The Eduserv Education Cloud provides compute and storage cloud IaaS for UK Higher and Further Education sectors. Eduserv was recently nominated as one of the top 20 tech companies to watch in 2012. Research Programme Director Andy Powell will describe the service and suggest ways in which cloud infrastructure such as Eduserv can support and enhance mobile technology projects.

Andrew Betts, founder of the web technology firm Assanka, will talk about the HTML5 web app they developed for Financial Times. HTML5 includes features that allow the creation of web applications that work while offline, and Assanka’s FT web app demonstrate how this capability provides excellent user experience in conditions of intermittent or no network access.

These are just a few of the projects that will be showcased at Open Source Junction 3. The programme also includes specially designed small group interaction sessions to help delegates understand each others’ value propositions and identify partnership opportunities.

Social activities on the evening of the first day, including a tour of the five century-old Trinity College, drinks at the historical Turf Tavern, and dinner at a surprise restaurant, will also offer excellent opportunities to network and further discuss collaboration.

Places are still available but early bird registration ends soon. We hope to see some of you at the event.

OSS Watch, in collaboration with Open Directive, are providing licensing, development, community and sustainability support to the project, which is now getting very close to a new release.

Developers have frozen the code and are preparing beta versions of DataStage and DataBank, which will be available for testing as virtual machines. Please keep an eye on the project website and twitter channel for updates on when and where you will be able to access them.

We will hold a launch workshop in Oxford on 2 March with colleagues from the VIDaaS project, who are building an exciting cloud-deployable Database as a Service system.

Attendance is free but places are filling quickly, so book early to avoid disappointment.

Emerson’s non-quote does raise a question, however. How do we identify technology which is better? With mouse-traps there are some fairly obvious metrics relating to mouse mortality and cheese preservation, but not all inventions are as easy to benchmark. The last few weeks have seen anouncements of upgrades to the world’s two most commonly used smartphone operating systems: Apple’s iOS (version 5) and Google’s Android (version 4). Each brings a raft of new features, although in both cases it has to be said that these new features are no longer as core to the operation of the device as innovations in earlier versions. Voice-operated search and facial recognition are nice, but hardly essential elements of a mobile computer, at least for now. Perhaps lost in the combative comparisons deployed by proponents of each OS is the fact that a genuinely key ability – web browsing – is implemented on both platforms using essentially the same code: the Web Kit open source project. While newer functionality is added by Google and Apple to differentiate the competing products, it pays them to cooperate on key, unavoidable elements of their offerings. Given this, it’s fair to repeat the question – how do we identify real innovation? The newer differentiating features appear to be the cutting edge of endeavour, but their very newness is a demonstration that – up to now at least – they have not been essential elements of the technology in question. Some of them will die away despite their novelty, having never truly improved the invention that they embellish. Like a cheese grater on your mouse trap, it’s possibly a nice idea and undoubtedly novel, but how useful is it really? Only time will tell, and in the meantime better springs, and better browsers, are being developed.

So perhaps the question needs to be: “looking back at innovations that have proved to be key, how do they tend to develop?” Using the answer to this, we might be able to form some techniques for looking at our cutting-edge-but-possibly-pointless innovations and making guesses about their eventual utility. We might even be able to identify over-arching strategies for conducting and rewarding innovation…

Here we get into an argument that flared up earlier this month, when a video of Francis Gurry, the Director General of the UN’s World Intellectual Property Organization (WIPO) back in June was discovered by the internet commentating community. Gurry was speaking to sum up his views on a debate which had just taken place on ‘Accelerating Growth and Development’ in relation to invention and intellectual property. Gurry’s argument was seemingly  summed up by the headline on the BoingBoing article which drew it to the internet’s attention: “WIPO boss: the Web would have been better if it was patented and its users had to pay license fees”. Reading the article, though, even the quote that BoingBoing had pulled failed to use that emotive word ‘better’:

Intellectual property is a very flexible instrument. So, for example, had the world wide web been able to be patented, and I think that is a question in itself, perhaps the amount of investment that has gone into or would be able to go into basic science would be different. If you had found a very flexible licensing model, in which the burden for the innovation of the world wide web had been shared across the whole user community in a very fair and reasonable manner, with a modest contribution for everyone for this wonderful innovation, it would have enabled enormous investment in turn in further basic research. And that is the sort of flexibility that is built into the intellectual property system. It is not a rigid system.

Reaction to the video from proponents of open content and open source across the internet was voluble and aggravated. Gurry was accused of being ideologically indoctrinated and blinkered, tied to anachronistic models of IP registration and exploitation even in the face of the incredible growth and success of the web largely without the intervention of these models. In fact though, the most that Gurry says is that the web would have been ‘different’. Taken in the context of the statements which preceded it (and which you can hear by downloading the video), in which the value of the traditional IP systems had been questioned repeatedly, Gurry’s statements do not really support the distillation they were given, and which caused so much anger. He is trying to argue that the web could have grown within more traditional licensing structures. Whether he is right about this or not, he is not claiming here that it would have been ‘better’ under those circumstances.

The anger and confusion here are natural, though. The battle lines between proponents of the traditional and the more ‘open’ approaches to innovation (and here we should note that the buzz phrase ‘open innovation’ often itself refers to deeply traditional IP exploitation patterns) have long been drawn, and the forces on both sides are keen to tackle and destroy the arguments of their opponents wherever they see them. The web is often perceived  - with much justification – as a triumph of innovation outside the traditional IP exploitation framework. To hear someone perceived as being part of the old-guard even discussing it can seem presumptuous to some ears. Yet in reality the implied dichotomy here is simplistic. The open licensing movements themselves are underpinned by the arcane operations of traditional licensing and exploitation. While they may give these operations an innovative twist, they could not be enforced or defended without them. Conversely, Gurry’s example of why  the patent regime is beneficial fails to address the criticisms of openness proponents. He points to the publication framework implicit in the current patent system, and makes the comparison between the saxophone – which has fully documented design documents available thanks to its having been patented – and the violin – where many secrets of producing the greatest instruments have been lost through secrecy and the passage of time. This critique – while interesting – is almost wholly inappropriate as a defence of the current system in opposition to more open models. In the modern case, both models involve complete publication – the distinction lies in how benefits are reaped from exploitation and by whom.

Given the frequent failures of either side in this debate to engage with what the other is actually saying – illustrated by this sad tale –  it’s not surprising that telling which innovations are better remains hard. While ideology is important, it can often obscure our view of what actually matters to most people: how many mice are killed (or indeed captured).

Jay Lyman, Senior Analyst at The 451 Group recently twittered that the “overwhelming message @ Oscon so far is open source now driven mostly by innovation.” This in turn prompted Brian Proffit of IT World wrote:

if you put yourself in the right mindset, this kind of innovation is easier to spot than ever. Open source isn’t “just” the alternative anymore, it’s becoming the mainstream technology in a lot of areas.

When we created TransferSummit the objective was to highlight the fact that open source is open innovation in software. However, unlike most open source related conferences we are not interestedin preaching to the converted. Open source and the innovation it brings is important to everyone and TransferSummit is designed for everyone, no matter their existing level of open source knowledge.

TransferSummit 2011 has the same goal and has an equally impressive range of speakers for you to engage with and share ideas. We have:

• 2 days’ trainings
• 2-day conference across three tracks
• Mobile focus, virtual track
• Government summit
• Hands-on learning and break-out sessions
• Gadget playtime
• Gala dinner
• On-site housing
• Intimate setting, limited to 300 participants

This is the must attend event for 2011, tickets are selling fact so register now.

Stephen Walli, the Technical Director of the Outercurve Foundation attended both events and, during the internal evaluation phase in which OSS Watch consider how to improve the next event Stephen kindly took the time to provide some really valuable feedback. This post attempts to encapsulate this feedback and frame it in the context of the work OSS Watch are doing, alongside others in the sector. I’d like to thank Stephen for giving me permission to quote large sections of his emails to the OSS Watch team.

Good software is developed by good software developers.  It involves a discipline not found in most developers.  Rigorous version and config management, checklist for style and review, desk checking reviews before commits, automated builds, fully automated tests are all necessary steps to successfully, reliably delivering binaries that work, and most importantly when a binary fails you know exactly how you created it from what tool chain (and conceivably what version of the tool chain) and source using what version of the recipe so you can reliably rebuild the failure and fix both it and the test environment.   Great developers may be great architects, but every great developer is a good disciplined developer first, and they really don’t know how to build software otherwise.

Personally I couldn’t agree more. I have long argued that we suffer from a significant lack of good software development discipline. As my Software Engineering lecturer told me, far too many years ago, “I’m not going to teach you to build software systems. Even a modest software system is as complex as a small town. I’m going to teach you how to make a dog kennel, or if you are really good a shed, we don’t have time for more than that.”

I, like most students, was taught the waterfall method of software development, a method that wasn’t in common use when I was a student and is in even less use now. Furthermore, most students leave University without any knowledge of version control, testing, continuous integration, build tools, issue tracking or any of the other disciplines Stephen wants to see in a good developer.

In general most students graduating from university have been taught how to build a kennel and are unaware that to build a tower block they need cranes, forklifts, jack-hammers and many other tools. These are the tools that we have to learn about when entering our first real software development team. In such teams, if we’re lucky, there will be one or more developers who understand the importance of the correct tooling, rigour and discipline in software engineering.

I’ve been fortunate enough across the ~25 years I was around software development projects (IT) and products (ISV) to always have at least 10% of the organization and sometimes as high as 30% be such good developers.  The rest of the team had an easy time contributing appropriately because of the culture at the top set by the few.  We all know what needs to be done — just few of us do it.

Stephen goes on to observe that he is not one of these developers. Sure he can develop software, he knows how to use the tools and he knows the discipline needed in good software projects, but he looks to others to enforce the necessary discipline on his work, and that of others (for the record, so do I).

Without such discipline at the top, I believe no project can succeed regardless of whether it’s academic, IT, or government.  This is why Apache and Eclipse and Linux have such well defined and documented engineering disciplines around them, and why they succeed.

I’m lucky enough to have worked with Scott Wilson at CETIS on the Apache Wookie (Incubating) project. Scott is a visionary and he loves to understand and work with open standards. Like me he loves to bring people together and get them to collaborate, this makes him the visible driver of the Wookie project. I’m there to mentor the team in the Apache Way, to ensure they understand what needs to be done. However, neither of us are the ones who are moving Wookie towards being a truly great software project. A project that is attracting commercial engagement. Sure, we have our strengths, but neither of us are the disciplined software engineer needed to ensure the success of the software itself (which is different to the success of the concept hawked by Scott).

Over beer the other day Scott and I were both happy to sing the praises of Paul Sharples on the Wookie team. Without Paul and his rigorous attention to detail Wookie would not be on the verge of it’s first official Apache release. Without Paul we would not have the  rigour needed to go through the various IP and quality control processes of a good open source software project. Without Paul, Scott and I would probably be telling everyone how great it is whilst gradually making the code decay with our endless hacks.

Scott and I raised our glasses to Paul and all the good software developers out there. It is my hope, and I believe that of Stephen’s, that OSS Watch will continue to seek out teams that have a “Paul” in them as well as a “Scott”, through those teams we can create true academia-industry links that will benefit all.

This second event in the Open Source Junction series targets specifically context-aware mobile technologies. Speakers from industry and higher education institutions will present their most recent work and will explore opportunities for collaboration in this area.

For more information about sessions and speakers, and to register, please check the OSS Watch event page. Early bird registration ends tomorrow.

It gives me great pleasure that one of my own hires, Sander van der Waal, has agreed to be my successor. Sander has been a key part of the team for some time and is well equipped to take over from me as manager. I have no doubt that Sander will continue to deliver, through the brilliant OSS Watch team, an advisory service designed to ensure our sector benefits from, and contributes to, open source.

Before explaining what I’ll be doing next I want to explain why I’m leaving.

Difficult Times

The academic sector is changing. It’s changing in fundamentally important ways. Most importantly, for OSS Watch, the funding models for services like ours is changing. For some time I, as manager, have been seeking to ensure all staff within the OSS Watch team have a level of job security that our current funding model is unable to provide as we move forwards.

As background consider that OSS Watch staff, including myself, have been on rolling three month contracts for nearly a year. Naturally, under these circumstances it has been very difficult to provide a valuable service to the sector. I’m extremely proud of the OSS Watch team and what we have achieved under these circumstances.

I’m not interested in complaining about the lack of security for our staff. The reality is that the economic and political environment of the last couple of years is the root cause of these difficulties and it’s pointless to suggest that the problems could/should have been avoided. The fact is OSS Watch is a minuscule part of this whole sector and nobody is to blame for how things have affected us. In fact, rather than complain I would rather state, loud and clear, I have nothing but praise for those with a direct influence on our work.

The JISC, through our programme director Matthew Dovey, have done everything in their power to ensure OSS Watch remains viable. Similarly our host, Oxford Universities Computing Services, have been faultless in their handling of staff contracts. There were a number of times that OUCS could have legitimately cancelled our employment contracts, but this was never seriously considered. On behalf of the OSS Watch team I want to thank the JISC and OUCS for taking responsibility and not sweeping important services like OSS Watch under the carpet.

Similarly, I want to thank the OSS Watch team. Despite extremely difficult circumstances the team have stuck by me and my plans for the future of OSS Watch. This is a testament to both the value they place in OUCS as an employer and to their commitment and contributions to open source advice in the academic sector.

So, if I am so happy with our funders, employers and team why am I leaving?

An OSS Watch Spin-Out

During the last year I and the team have been working to secure additional funding streams for OSS Watch. We’ve had plenty of success in attracting funding for specific support activities in key projects, this has enabled us to significantly reduce our demands on the JISC. In addition, I’ve had success in generating interest in an OSS Watch like service for the private sector.

As a result of this interest from the private sector I’m leaving OSS Watch in order to start a new company, OpenDirective, with Steve Lee as a partner. Our goal is to connect the smart folk in the UK research domain to the people who can take their software developments to market.

Both Steve and I will remain engaged with the OSS Watch team to ensure our services are complementary.  Through our collaboration we will seek to identify opportunities for technology transfer and, in so doing, generate alternative streams of revenue for open source activities within the academic sector.

What Does This Mean to You?

First and foremost, OSS Watch continues to provide free, at the point of use, advisory services to the UK Higher and Further Education Sector. OSS Watch remains fully independent of commercial interests and as such continues to provide unbiased, non-advocacy advice. However, these services will remain advisory only. That is OSS Watch will not tell you the best course of action, nor will they actively engage with the implementation of your chosen route. OSS Watch seek to enable you to choose and act upon the right option.

OpenDirective, on the other hand, are able to provide more proactive and engaged services. We won’t just advise you of your options, we’ll be willing to tell you what you should do. We’ll even help you do it. Naturally we’ll explain our reasoning, but we’ll be spending more time on making your project succeed than deliberating over all possible alternative actions. OpenDirective will not be free at the point of use, but our contracts are performance based, that is we’re happy to put our time and money where our mouths are.

Interestingly, one of OpenDirectives first clients is OSS Watch. For the foreseeable future, Steve and I will remain a part of the OSS Watch project support team. So, if you are part of a project that has Steve or I are assigned to then things will not change a great deal for you. At the same time, a couple of our initial contracts are injecting funding and community development work back into OSS Watch projects.

Taken together we believe that the combination of OSS Watch and OpenDirective give you a new range of options that should enable you to reach your goal state in the most appropriate way possible for your unique position.