Open Standards are not enough to prevent lock in

Many people claim that open standards are the answer to lock in problems of software. Even our government can be heard to claim open standards are the answer:

There can sometimes be a danger of lock-in with some proprietary providers, and we must avoid developing an over-reliance on individual suppliers. The Government, via the Office of Government Commerce, work hard to avoid that by using open standards to ensure that different suppliers’ software can be used interchangeably. (Angela Eagle, The Exchequer Secretary to the Treasury during a parliamentary debate)

However, there is much more to the lock in problem than the format the data is stored in. We also have to consider how this data is stored and processed in any given business process.

Until recently I’ve been reading about peoples concerns over the closed nature of Microsofts Sharepoint with a pinch of salt. I have to admit I just didn’t get the problem. If the data was in an open format you could just take your data and run, right?

Well no, that is not the case. Thanks to my old boss Randy Metcalfe, I now realise the lock in comes in the form of business processes tied to the repository. Matt Asay explains in an interview with lwn.net:

Let’s assume you store data in ODF in a Sharepoint repository. It doesn’t matter that ODF is an open format. The repository holding it is proprietary, and that proprietary lock-in is doubled by the fact that the enterprise will build (proprietary, non-standard) workflows to manage that content which keeps content a prisoner to Microsoft.

This may be true, but the fact is that Sharepoint makes it possible to build these workflows. I’m aware of no other single tool, open or closed, that is as complete. Almost certainly this is why many of the people I speak to in the education sector report an interest in Sharepoint.

What worries me is not that these people are considering Sharepoint, it’s that they think that a move to Sharepoint, coupled with an adoption of open standards will prevent a lock-in to a single vendor. This does not appear to be the case.

So, if you are concerned about vendor lock-in what can you do?

Firstly, you should recognise that no software tool can be rolled out across an organisation without significant configuration and optimisation for the (often fluid) local business practices of that organisations. Buying any off the shelf product will always result in the need to also buy consultancy and/or staff training to provide ongoing support . As a representative of a major UK university recently told me “we thought we could buy the licences and pay some consultants and that would be it. Unfortunately it’s not as simple as that.”
Secondly, we must recognise that it is possible to create a software stack using mature and successful open source software that will do everything Sharepoint will do, and more. Sure, it takes effort to do this, but it can be done.

Finally, we must ensure that we evaluate any closed source solutions against any open source alternatives, taking into account all strategic, technical and resource objectives.

Why must we consider open source? There are many reasons, the most relevant to this post is that open source, coupled with open standards prevent lock-in.

What are open source and free software?

I have been noticing in my LUG‘s mailing list that some people (even the geeky linux-friendly sort) have a hard time defining “free software” and “open source software”, and sometimes take this topic as if there was a good vs. evil war going on.

“Free software” and “open source software” are notoriously loose terms

The advocates of “open source software” tried to make it a trademark, saying this would enable them to prevent misuse. This initiative was later dropped, the term being too descriptive to qualify as a trademark; thus, the legal status of “open source” is the same as that of “free software”: there is no legal constraint on using it.

Free software aficionados are usually quick to point out that ‘free software is software that fulfills the 4 freedoms in the Free Software Definition of the FSF’.

Recently, somebody wrote to the aforementioned mailing list saying that software is free or not irrespectively of what the FSF says. It just needs to fulfill the 4 freedoms. But then again, who decides whether the 4 freedoms are fulfilled? The problem is that when you say ‘This program is free software’, as Bill Clinton famously put it (in a different context),

It depends on what the meaning of the word ‘is’ is.

That is, the 4 freedoms are not a mathematical expression that can be evaluated unequivocally. They are not even in legal language that can be argued in court (as my colleague Rowan noted). Something similar happens with the Open Source Definition of the OSI.

Somebody replied in the LUG’s mailing list saying that all you need is to ask a lawyer who knows about licences, and he or she will tell you whether the 4 freedoms are fulfilled. But this is not good enough, obviously, as different lawyers may have different opinions, and as I said before, the 4 freedoms are not in legal language.

What is more, not even the FSF thinks that the 4 freedoms are a perfect expression of the idea they have about “free software”

Finally, note that criteria such as those stated in this free software definition require careful thought for their interpretation. To decide whether a specific software license qualifies as a free software license, we judge it based on these criteria to determine whether it fits their spirit as well as the precise words. If a license includes unconscionable restrictions, we reject it, even if we did not anticipate the issue in these criteria. Sometimes a license requirement raises an issue that calls for extensive thought, including discussions with a lawyer, before we can decide if the requirement is acceptable. When we reach a conclusion about a new issue, we often update these criteria to make it easier to see why certain licenses do or don’t qualify.

So if we cannot rely on the Definition of Free Software, does this mean that we cannot define “free software” at all? In fact we can, if we accept that free software is software released under a free licence. In this case, free form language gets hammered down into the legal mold, and a lot of ambiguity is removed.

Of course the burden is now on deciding which licences are free and which ones aren’t. Accepting that free licences are licences that the FSF say are free seems to cause a lot of discomfort to some people.

This is not so much of a problem with the open source community, who seems more willing to accept that open source licences are not those that are believed to fulfill the Open Source Definition, but those that the OSI certifies are open source (and that’s the approach we follow in OSS Watch too). The OSI has even registered the “Open Source Initiative Approved” trademark for specific software products.

I guess that the reason why giving the last word in terms of “free” to the FSF causes discomfort with some people is that the free software community is built on the idea of freedom as a paramount value, not only for software but for society as a whole, and that subordinating “free” to the FSF is giving away part of that freedom. In the end, “free” would be a badge awarded by an opaque team of lawyers.

At the same time, I don’t think that anybody would seriously consider as a better option to have any number of licences, each of which needs to be evaluated by each individual in order to decide whether they are free or not.

Both the OSI and the FSF have approval processes in place to decide not only whether a licence fulfills certain requirements, but also to make sure that it is not similar to an existing one and thus cutting down licence proliferation.

Licence proliferation is the enemy from within for open and free software, because it is possible to have licences that fulfill the open and free definition, but at the same time prevent different projects to combine their outputs and collaborate. The most significant example being the incompatibility between the GPL v2 and Apache License v2 (the GPL v3 is compatible with Apache Licence v2, though).

I believe that at this point the need to standarize licences, make them compatible and reduce their number outweights the risk of the FSF going awry, and hence I’m happy with the statement “Free software is software released under a licence approved by the FSF”.

Open Development in Football

Open development is spreading into unexpected areas…

For the first time in football history, fans have the opportunity to buy and
then take control of a professional football club – both on and off the pitch. Every MyFootballClub member will have an equal say in team selection, player transfers and the running of the club.

I’ve been wanting to blog this since My Football Club announced their first purchase, but other things have been getting in the way so I’ll leave the analysis to the reader.

Changing Licences

There’s a story in wired about licences in flickr photos. The problem is that flickr requires users to tag each photo with one of a range of licences (including “all rights reserved”). Users can change the licence at will, either on individual photos or on thousands at once.

If a third party takes a creative commons licensed image, reuses it under the terms of the licence and the user subsequently changes the licence on the image on the flickr site, difficulties arise.

The creative commons licences are perpetual, containing words like:
Subject to the terms and conditions of this License, Licensor hereby grants You a worldwide, royalty-free, non-exclusive, perpetual (for the duration of the applicable copyright) license to exercise the rights in the Work as stated below:

So the third party can continue using the image under the creative commons licence indefinitely, provided they have a local copy. The user (the copyright owner) has now removed their offer of the image under the licence, so proving they are entitled to use the image could be problematic, unless they’ve done their homework and kept some form of log of the licence. Getting a new copy of the image under the old licence if they haven’t kept a copy is likely to be impossible.

I have no idea what happens in the case where an image is pulled dynamically from flickr and built into a composite in a way which breaches the new licence. Presumably such dynamic system need to check the licence every time, as is entirely possible using the flickr API.

The take home message? Keep track of what software and content you’re reusing, keep and archive a local copy of everything you use.

Is Open Social an open standard?

The initial flurry of activity over open social is over. Now people can settle down and consider its merits and its warts. This very topic came up yesterday in a session I ran on open development at UKOLN. We discussed isues such as what makes it a standard? and what makes it open?

The initial open social partners will have us believe that it is a standard because they say it is one (a de facto standard). They also tell us it is open because anyone can implement it. But why should we listen? How can we influence its design?

The people behind open social are not stupid. They are also realistic enough to know that open social will only become a genuine de facto standard if it is used beyond the initial group. The easier it is to implement the more likely it is to be used, the more it is used the more “standard” it becomes.

Accordingly, Ning have offered to donate their implementation of open social, complete with an initial set of committers to support it, to the Apache Incubator. Assuming the project is accepted this will result in an easy route to implementation for almost any project.

So that’s initial take-up pretty well sorted then (unless the users think open social is a bad idea of course). What about the “open” part?

The ASF has a history of managing reference implementations of standards and of protecting users in standards definition processes. They are often held in high regard for this important work. I hope this means that the open social partners are going to allow the ASF to create a valid user community around open social. That is a community with a voice and an ability to influence the standard.
Lets see what smart developers can do with this. If you want to turn your project into a container you probably want to save yourself lots of effort and wait for the Ning donation to hit the Apache Incubator.

Open access bill in the USA looks likely to pass

Open access looks about to pass a significant milestone with a bill in USA Congress which requires open access to National Institutes of Health (NIH) funded outputs. While NIH funded research is only a small fraction of the peer review funded research globally, it’s one of the largest coordinated research programs with huge inertia, both externally and internally. It seems likely that all significant medical and genetics peer review publication forums will be open access in the near future. NIH also funds work in a whole range of disciplines which impact on human health, so they’ll receive an open access boost too.

Such a big win is built on the work of a whole lot of individuals and groups world-wide, including Stevan Harnad (long term open access über-evangelist) and the JISC funded Sherpa, OpenDOAR and ePrints projects. Congratulations guys.

Score one for JISC open development

I started working for OSS Watch in January of this year. I was brought to the team to supplement their already extensive knowledge of all things open source with my own take on open source development outside education.

After a short period of bedding in I was sent out, with other members of the team, to consult with projects. These consultations continue today and are probably our most effective support tools. Almost ten months into my involvment with OSS Watch I’m starting to see genuine results from some of the early consultations.

WebPA, largely driven by Nic Wilkinson, have been steadily building a community support structure for their project. In the last few weeks things have really started to take off for them.

Here are some annotated quotes from the project blog, I think these quotes show how the project has developed and could (make that should) be used as a model for all projects wishing to go an open source/open development route.

I never imagined that choosing the potential licence that we would use could be so difficult. This is all leading up to the writing of the consortium agreement and the future open source release of WebPA. 2 January

Sort out our licence and IPR position early.

We have also managed to get some very basic presentations together as an introduction to the project…. Our other major mile stone is getting a JISCmail list. 17 January

The WebPA Project has started to disseminate further afield in ‘DLib’ magazine. 8 February

Make sure your potential community is aware of you and can communicate with you from day one, sure they won’t come flocking to your door yet, but creating a brand is the first step.

Yesterday we had a very informative meeting with Ross and Rowen from OSS-Watch. 7 March

All this to do and two open source books to read! 23 March

Do your research, building a community led project is not hard, but it is quite different to any other kind of project structure and it won’t happen overnight for you. Understanding best practice and figuring out how to apply it to your work is the key to success.

As with everything related to agreements and IPR, there is a distinct lack of understanding as to the amount of work that needs to be completed. There is also a distinct lack of understanding as to why this can be so important to the project and its potential success. 24 April

Recognise that you don’t know everything and get advice early (yes that is another shameless OSS Watch plug ;-)

Just to make sure that we are getting our moneys worth out of OSS-Watch (not that we pay) I asked some colleagues here at Loughborough if they would like to join the WebPA team for a couple of hours and sit in. Everyone who attended to find out about Communities has given positive feedback. People have said that they wished they had been able to give more time to attending the afternoon session, to find out more. 10 May

Always help those who help you and give credit where it is due (thanks Nic, I’m sure none of our readers realise I’m quoting this one merely to plug OSS Watch ;-)

It has also highlighted to the WebPA team some of the routes that the project can take as exit strategies or (a better term) project sustainability from when the funding runs out. 10 May

Plan for sustainability as early as is practical. OSS Watch encourage you to consider it during the bid phase, we are, of course, here to help with this. The sooner you start planning for it the more likely it is to happen.

However, this is now giving me a headache! I have made it overly complicated. 15 May

Keep it simple. Make it do what users want it to do and nothing more. Keeping it simple not only helps you, but it also helps anyone coming to the code as a potential contributor.

After a fairly swift poll of the project partner and the other potential pilot the opportunity to use LDAP arose…. The first problem I encountered was… 25 May

Be open about issues you are facing and how you intend to address them. By now you have a small audience, someone out there may have the answer, or may stop you making the wrong decision.

The only concern I have here is that WebPA are doing this in their blog. It should really have happened in their mailing list since the intention should be to encourage contributions of all kinds. People can only comment on blogs, they can’t create new posts so their contributions are limited to the topics you choose, what if someone wants to ask an unrelated question? You need to create a culture of posting to the publicly archived mailing list.

But hey, I’m not going to criticise – doing it in the blog is better than not doing it at all (and Nic was also posting regularly to the mail list as well).

who is our community and what exactly will the wiki be used for? 7 June

Don’t use technology simply because it is “cool”.

a wiki is not the correct tool for discussion to be carried out on. We have a mailing list, which anyone can join and the achieves are openly available. Hence the mailing list is the correct tool and a wiki an incorrect tool. 7 June

Use the right tool for the right job.

In this process we hope to get more institutions to host WebPA for their academics. From this we hope to build a community of users. 15 June

Look after your users first, without your users there is no project. From your users your contributors emerge.

Yesterday I attended a workshop run by OSS-Watch on building communities. For us as a young open source project it was really useful to get the opinions of the community that we work within but never really think of as a community! 21 June

Anyone tired of the OSS Watch self promotion in this post?

This leaves one route to go down, which is to change attitudes. Whether this is the best course of action only time will tell 21 June

If current practices are failing and you think you may understand why speak loud and clear. Don’t be arrogant, don’t tell “them” they are wrong, just lead the way and those who agree will come and help. Those who don’t agree will challenge you and will help you ensure that you have considered all options, thus you are more likely to choose the right path.

my next delve [into Sourceforge] was to set up a list for feature requests and set up the few other lists for people to use when the project moved forward again. 6 August

You absolutely must have an infrastructure that allows your community to engage with you. It doesn’t take long to do and once done it actually makes project management much easier, even if you don’t have a community yet.

there are people out there trying to change both the system and the cultures, mainly thanks to the work of Randy Metcalfe, and now Ross Gardler, and the lads [and ladies] at OSS-Watch (http://www.oss-watch.ac.uk/). Due to them I no longer feel isolated as they have set up a community which is thriving. 14th September

HeHe – Nic really knows how to get the best out of us at OSS Watch.

Have we a break through, have I finally stopped talking to my self on the JISCmail list? 21 September

Building a community is slow, it can be frustrating, it can be lonely, but it is worth the effort. Keep reading…

I am in the process at the moment of supporting a number of institutions with their own installations of WebPA. 8 October

I released a version of WebPA as a download on Friday 5 October 8 October

I have in the past twenty four hours been asked about the projects plans to develop integration modules for a particular VLE. 16 October

The road map document lets the projects community members see where the project is intending to go for the next phase of work… In order to ensure that the community needs are reflected… 22 October

Understand your users, engage with your users, satisfy your users, make it easy for users by providing early downloads. Without your users your project will not be sustainable, the more users you have the more likely you are to find a sustainability route.

This leaves me to add the information to the trackers myself. At the moment this is okay, as there are not to many requests. I am encouraging the users to use the sourceforge system, so every time I email back with the solution I endeavor to include tracking information, as well in, the hope that at some point they will use the system. 8 October

As the number of users grows, so will the demands on your support activities. You must train your users to use the proper channels so that the support load can be spread across the community.

Remember users are usually the best people to support other users. Create a culture in which users who got free support from you are willing to give free support to others. Failure to do this means that you will eventually become a victim of your own success and will never have time to do development work.

it [driveby contributions] is a way of building a project where people can contribute the small element they need to and then leave the project. Unknowingly I facilitated this type of action 26 October

WooHoo!!! A contribution from a third party – well done WebPA – all that hard work on community development is starting to pay off.

Speaking personally, I had taken this kind of contribution for granted until I read this post. It just hadn’t occured to me that people didn’t realise this is the most common kind of contribution in an open source project. I’ll make this explicit in all my future work.

If you don’t undertand why these small contributions are important consider this WebPA case. It was a small bug that would only occur in a specific configuration in the authentication system so it was unlikely WebPA would find it.

If this bug went unfixed, how many potential users would try to use the software but give up because they couldn’t log in to their initial installation? Since every user is a potential contributor, each lost user is potentially a lost contributor. Furthermore, each lost user could be a lost paying customer for Loughborough or A.N.Other should they offer a paid product or service based on WebPA. In other words each lost user results in a decrease in your chances of reaching sustainability.

These small fixes are the lifeblood of an open source project not only because they ensure a higher numbr of satisfied users, but also because having had one patch accepted the contributor is more likely to submit another, then another and another. Eventually you have a new developer to vote into your project and you are on your way to sustainability.

One piece of advice we were given was to make a demonstrator and make it available to potential users to see what the software is about. Well this was realised at the end of October. Within this first week we have had a phenomenal response. 1 November

Make it as easy as possible for users to evaluate your software. If someone tries an online demo and likes what they see they are far more likely to spend the time downloading and installing your software. If your software is not a webapp then create a series of screenshots and/or screencasts (actually they are useful for webapps too).

We are now up to date.

WebPA has done an incredible job of building a community support structure around their code, they are even starting to see genuine community activity.

Will all this effort make the project sustainable?

I don’t know, it is far to early to tell. Building sustainable communities takes a long time. For example, see this graph of activity on The Apache Software Foundations mailing lists which shows that it took around four years before the Apache community took off.

Although it is too early to say whether WebPA will reach sustainability I will say this: if the users continue to appreciate the value of this software and the WebPA team continue to proactively support them in this way the chances of reaching sustainability are very high since all options are now available.

WebPA I salute you.

If you think your project could benefit from OSS Watch’s advice please contact us – we are JISC funded and so won’t cost you anything (UK HE and FE only I’m afraid).

The end of “walled garden” social networks?

I worry about attempting to build communities in walled gardens.

Is OpenSocial going to change the world? (If you need a quick intro to OpenSocial then watch this short video, or if your prefer reading try the API site).

Now all we need is a set of open source social network tools that allow us to build the niche social networks we need for real work whilst not expecting the participants to be totally cut off from the larger (and some would say more fun) networks that are currently popular.

FOAF + OpenID

FOAF + OpenID is a semantic web attempt to solve the problem of blog spam. The idea is that those people who have FOAF files and OpenID identities can identify each others networks of friends, colleagues and acquaintances using their FOAF files and authenticate the individuals using OpenID.

I’ve found a flaw in this: I could have (and probably should have) a link in my FOAF file to a semantic wiki representation of myself, which is (in the way of wikis) world writeable. Spammers could easily edit the wiki to insert a link from myself to them which would let them become part of the group and spam us.

There are a number of fixes for this:

  • Check the metadata in each FOAF file to ensure that it claims to be written by the subject of the file (which wouldn’t be the case for the wiki). This would require many FOAF/RDF generation tools to be updated.
  • Add trust attributes to external links in FOAF files. This would also require many FOAF/RDF generation tools to be updated.
  • Compile a list of known world-writable RDF sources and use it to black-list them. This would always be playing a game of catch-up and there some sites might slip through.
  • Require trusted users not to link to world-writeable RDF sources (or sources of RDF that harvest from the wider web). This requires that the semantic web workers work in a walled garden and not link outside it into the wider web.

None of these are easy.

Somehow this whole thing reminds me of the OpenPGP web-of-trust, without the cryptographic underpinnings.