At OSS Watch we periodically review all the resources on our main website to make sure they’re accurate and up to date. Last week it was time to revise our case study on Apache Wookie, which is a project I’ve been involved with for some time.

OSS Watch became involved with Wookie while I was working in an EU project based at the University of Bolton. The project as a whole had done lots of interesting stuff, but as with many large projects the whole was somewhat less than the sum of its parts; the central joined-up platform wasn’t really going to take off after the project finished. However, in the process we had built quite a promising system for adding functionality to the core portal shell using the W3C Widgets specification.

Towards the end of the project I went to an OSS Watch event, and spoke with Ross Gardler about what we were doing. Ross explained the Apache Incubator model to me, and from there on I was hooked.

Fast forward to 2013, and Apache Wookie is out of the incubator and a top-level Apache project, and is now on its seventh official release (the last one was in April). Its not a huge project – the team is still small, though its far more diverse than when we started out.

The tempo of development has also slowed in recent years. However, in part thats due to the maturing of the software to a point where code churn for its own sake has a negative impact on the projects that depend on it. Most recent updates have been fixing bugs affecting deployment in various unusual configurations, driven largely from reports by users. So this isn’t necessarily a bad thing!

Something that has also had a very positive impact on the project is having a very active downstream project – Apache Rave. This has driven a lot of improvements to Wookie to improve integration and deployment.

Two major EU projects have been working with Wookie and Rave over the past two years, and are coming towards their end – one this year, and the other in 2014.

Unlike previous projects they have focussed on working with existing software projects rather than going it alone, and have contributed code, user studies and content.  This has been a great experience, and hopefully future projects can learn from this approach.

Wookie stands as an example of how OSS Watch can help take work from within the HE sector and turn it into a sustainable open source software project; and as a beneficiary of this approach I’m keen to offer the same help I received to others.

Do you think your University-based project has the potential to go further? If so, get in touch!

Read the updated case study on Apache Wookie at OSS Watch.

(Photo by Silus Grok, used under CC-BY-SA license)

One of the most annoying things that can happen to software is for it to become “abandonware”, particularly if its something you rely on.

This is where the original developers of the software cease further development or support, usually to move onto other projects, or sometimes due to a change of employment or other personal circumstances. Whatever the reason for it, the software that was once a key part of your infrastructure ceases to be updated, and over time its going to be less and less viable.

However, with Open Source there is sometimes the potential to “reboot” a software project even when the original developers have moved onto other things.

This recently happened to me with a library called HtmlCleaner. HtmlCleaner is a handy little library that, as the name suggests, cleans up “dirty” HTML and allows you to modify it programmatically. There are other libraries for doing this of course, but HtmlCleaner always had the edge for me due to its light footprint, small number of dependencies, and an API that provided the functions I needed. HtmlCleaner is used in Apache Wookie, for example, to clean up HTML in web widgets and inject additional JavaScript needed for extra features.

Unfortunately, the developer of HtmlCleaner, Vladimir Nikic, had moved onto working on other things, and the last release of the library was in 2010. Then recently we came across a bug in HtmlCleaner that was affecting Apache Wookie.

After some Googling I found that others had come across the same problem, and had forked the project to apply their own fixes. Which didn’t really solve the problem, as you then have to ask yourself which of the forks are viable and up to date, and also run the risk of fragmenting the community.  Wouldn’t it be better if we could get those interested enough to make a fork, to take an active role in the core project?

HtmlCleaner already has a large user community, and the project regularly receives new patches and feature requests; all that was missing at a minimum was someone available to thank users for their contributions, review and apply patches, and manage the releases.

I got in touch with Vladimir, and Patrick Moore, creator of one of the forks, and we agreed to “reboot” the HtmlCleaner project, with Patrick and I getting maintainer access.

I applied a user-contributed patch to fix the bug that was vexing Apache Wookie users, and released HtmlCleaner 2.2.1. Patrick is merging back in the various fixes and improvements that he’d made to their fork, to be released as 2.4. After a gap of two years, HtmlCleaner is back in business!

Of course, in another year or two perhaps both Patrick and I will have moved onto other things too. But if we do, hopefully like Vladimir we’ll be able to hand over to new developers coming from within the community.

Some open source foundations and umbrella projects list dormant projects that can be rebooted;  for example, the Apache Attic is where Apache projects head when they are no longer active, and there are processes for allowing a new project team to restart activity, either by moving the project back into the incubator, creating a fork, or forming a new project management committee.

If projects have been dormant for too long, technologies and frameworks may have moved on so far that reviving them is not a viable option, but in some cases there can be mileage in reviving  one of these projects rather than starting from scratch, particularly if there is still a user community to draw upon.

 (image by _bianconero licensed under CC-BY)

Our colleagues over at CETIS have been doing a lot of work on analytics recently so I thought it was about time I had a go at using some of the tools available for analysing open source software projects.

I’ve used Ohloh for quite a while, which is a useful tool for getting a general picture of a project and its committers, particularly its canned “factoids” relating to the overall level of activity, amount of comments in the code, languages used and so on. At OSS Watch we also have some more in-depth tools and methods for looking into how a project is working. However, I was interested in how you might be able to visualise the activity of the community itself; the first thing I found is a very nice application called Gource.

Gource reads the log files from most source & version control systems used by projects, including svn, git and mercurial, and generates an animation showing the interactions between the people and the codebase – so every commit is visualised as a person moving around the code structure and touching the files.

You can also export the Gource output into FFMpeg to create a movie. Here’s one I made showing the history of Apache Wookie – from project startup all the way through incubation at the Apache Software Foundation – in about 48 seconds:

The actual command used to generate the movie looks like this:

gource -640x310 wookie.log -t 48 -s 0.01 -hide filenames,dirnames,usernames -o - | ffmpeg -y -r 60 -f image2pipe -vcodec ppm -i - -vcodec libx264 -preset ultrafast -crf 1 -threads 0 -bf 0 wookie.mp4

The movie shows the project starting off with one contributor laying the groundwork before being joined by one or two others, and finally towards the end there are parts where you have four or five people working together, some focussed on one part of the project, and some on another.

You can also make the movie much slower, and show the usernames of the committers and the files and directories being worked on to get a better sense of “who is doing what” in the evolution of the project.

Unfortunately, what we see here is only part of the story, as these are all the commits made to the source code, but not the messages on the mailing list, the issues on the tracker, and the updates to the websites. For Wookie, for example, much of the growth of the community is seen in contributions on the issue tracker and mailing list – only a few made the transition into becoming committers.

It would be great to get the issue tracker history from Jira, for example, and include that in some fashion. With Apache projects at least there is also often an explicit link between a Jira ticket and related commits. Given that Martin Hawksey has shown you can use Gource for visualising OER deposits into Jorum using Google Refine I’m sure its possible!

I also made a movie of Apache Rave using the same settings; Rave is a much younger project but started out with far more committers, so the same activity here takes around 20 seconds:

Finally, here is another video from the Gource team showing a few different projects, including Moodle:

Its a common dilemma facing a funded project: do you deliver the software you’ve promised in your workplan by building it from scratch, or do you work with an existing open source project? If you build it from scratch, you keep control of the direction of the software, you can plan it according to your project workplan and schedule, and you know how you can report the effort involved to your funders. However, you’re likely to struggle with making that software sustainable after the life of the project.

On the other hand, engaging with an existing open source project brings a different set of risks and benefits; you can benefit from the contributions to the codebase from the wider community, and the software is far more likely to be sustained in the longer term as it doesn’t rely on the project funding. However, the project consortium is only one group of contributors to the community, and so the software roadmap and release schedule is no longer under control of the project.

This was the situation faced by the OMELETTE project, funded under the European Framework 7 Programme. OMELETTE had a need for a mashup delivery platform – in essence, a web platform for combining web apps created by mashup tools into shared workspaces. Requirements had been written, and there was a workplan identifying when functionality had to be delivered. The obvious route would be to go and build the platform, and then make some justification in the exploitation plan as to how the partners would make the platform sustainable when the funding ran out. However, the consortium decided to take a different approach, and work directly with the Apache Rave and Apache Wookie projects, which already implemented many of the requirements identified by the consortia.

Again, another obvious strategy at this point would have been to create a private fork of the software. That would enable the consortia full control over the direction and delivery of the platform; the drawback is that a fork can fall rapidly out of step with the original project, creating additional effort in porting important bug fixes and new features into the fork. This was going to be a big problem with Wookie and Rave, as both projects were evolving rapidly. Rave in particular as a fairly new project was getting new features implemented at a rapid pace.

So the approach taken was to work directly with the communities on the core codebase wherever possible rather than to create a project fork. This meant that the project could set up continuous integration (CI) using the current trunk of Apache Rave, ensuring that mashups developed in the project could be tested with the latest running code.

One immediate benefit was that some of the key requirements of OMELETTE, that could potentially have required considerable development effort, were being implemented as part of the Rave community roadmap. For example, inter-widget communication, a core OMELETTE requirement, was implemented in Rave without needing any real input from the consortium at all; all the project partners had to do was make use of it in their web apps when it turned up in the daily build.

For other project requirements, a more proactive approach was needed. One feature that OMELETTE needed to implement was a way of connecting workspaces with an external repository developed within the project; this could have been done as an OMELETTE-specific module, however when the project partners discussed the functionality it was found that it could be expressed as a more generic feature for connecting Rave with widget marketplaces (for example, Edukapp and Widgr). The internal requirement was rewritten using external language (terminology can be an issue when working across projects) and submitted to the Rave community as a new feature request. Almost immediately, other community members began commenting on their own need for such a feature. (This is now being implemented by OMELETTE partners and contributed back to Rave).

Of course not everything in OMELETTE involves open development; some aspects of OMELETTE for one reason or another haven’t lent themselves to working with existing open source communities, and have been developed entirely within the project. Some other components are closed-source.  However the platform used to integrate the whole project is Apache Rave, including contributions from the project.

So could this approach work for other projects? I think it depends on a number of factors, not least the willingness of the project consortium to make a trade off between control and sustainability.

Realistically, most software developed in funded consortium projects is not sustained beyond the end of funding, although naturally every project claims it will do its best, sometimes for larger projects even setting up their own foundations. However, I think privately most organisations involved would admit that often such claims of sustainability are over-optimistic, and project officers and programme managers are I think increasingly skeptical unless the consortium members have the track record to back it up.

Loss of control might be a problem for consortia, particularly where there isn’t a close alignment between the consortium objectives and how the wider community sees the future of the software. In this case, OMELETTE were using Rave and Wookie in ways that were clearly complementary with the use being made by other members of the wider community, and so the control issue never arose. As a backup plan, if there was a requirement needed by OMELETTE that didn’t fit the Rave or Wookie roadmap, the project could have created a fork, but this hasn’t happened yet – and seems very unlikely.

It helped that in the OMELETTE consortium there was already a partner engaged in both ASF communities, in this case the University of Bolton. Developers from Bolton were able to bridge the gap between the consortium and the communities, bringing other partners to ASF hackathons, and introducing developers from other partners in the consortium to open development.

If there was one clear improvement that could have been made, it would have been to have conducted internal capacity building activity much sooner and with more resources; it can take a lot of mentoring and encouragement to get developers used to open development practices, and there can be plenty of opportunities – and incentives – along the way for going back into a closed way of working. If more effort had been made from the beginning in mentoring developers, the project could have been contributing actively to the community much sooner than it did.