Open source still suitable for business-critical systems

Open source is business-ready. Not just for smaller companies that are looking for a simple CMS, but also for business-critical systems with high-volume traffic. This is not really news, and the examples are well-known, such as the use of Linux at the London Stock Exchange. But an interesting report (pdf) by Jim Norton, former president of the BCS, recently reinforced this message and it is an interesting read. For one, the report had been sponsored by Amadeus, a technology provider to the travel industry. As Glyn Moody rightly notes, this is not a company that has an interest in supporting a particular technological viewpoint, but is focusing on providing real solutions.

Benefits of open source systems

The report lists benefits of open source that are relevant to different kinds of stakeholders. Specifically for enterprise customers, Norton notes that open systems provide:

  • Access to greater innovation – A self-reinforcing open community has emerged of shared research and development that customers of open source software can tap into.
  • Improved supplier responsiveness – Access to an open community allows much better response time for problems. A customer is no longer dependent upon one supplier to respond adequately and timely, but has a wide community of people that may help out when there are problems.
  • Enhanced system accessibility – There is a wide variety of systems that has been developed openly over the last few decades. This enables customers to use and continue to use systems if they have the required expertise available. Customers do not need to depend on supplier-demanded software life cycles.

When focusing more on end consumers, the following added benefits are highlighted:

  • Greater choice and depth of services – Norton argues that certain services that are available for open source have simply not been financially viable under the economic model of proprietary software
  • Democratisation of services – By building services on open source software, removing limitations such as volume-related fees, greater innovation is possible and enables much more sophisticated services to be developed. Simply put, because components such as web servers, search engines and development frameworks are available as open source services can be developed on top of this without the need to worry about these lower level components.

Open source rather than cloud

Norton also analysed the pervasive trend of cloud and concludes that specifically in the area of risk management, some steps need to be taken in order to make sure that systems and data are secure and safe in a cloud environment. Control and visibility of systems are much less clear and obvious in a cloud environment, and auditing processes need to be in place to ensure this is properly addressed.

Another challenge is the interoperability and portability of cloud solutions, which is still very much under development. Open initiatives such as OpenStack and Apache CloudStack develop interesting solutions and communities that address these issues.

Finally, the report provides an analysis to inform IT management how to make transition from closed to open systems. It addresses some of the real practical concerns IT managers may have and helps them understand how to harness some of the benefits of open source. One important part of this is the focus on staff and the need to train staff and make them familiar with the new system and understand the technology. OSS Watch has provided a number of resources that can help you understand open innovation in software. Feel free to get in touch if you would like to find out more.

OSS Watch refreshed

Some exciting changes have happened at OSS Watch this week, when Scott Wilson and Mark Johnson joined our team. This is wonderful news, since both Scott and Mark have great in-depth experience with open source in the sector.

I am very pleased to hand over the management of OSS Watch to Scott. Through his work, Scott has demonstrated time and again how to successfully apply open development to software projects in the Higher Education sector, with Wookie, Rave and more recently projects like Edukapp. Furthermore, he has great insights in the sector and the JISC community, through all of his experience via CETIS and other services. Similarly, Mark has some great practical experience by working in projects such as Moodle and within the Ubuntu UK community. His presentation ‘the line of code that could’, about how his first contribution to Moodle propelled him into the Moodle community, is truly an OSS Watch classic. It’s wonderful to have both of them on board.

With them joining the team, my own role at OSS Watch has changed, although I am not gone completely. A major reason for me to step down as manager of OSS Watch was related to my personal decision to leave Oxford, where our offices are based, to go and live in London. However, I will continue to work with OSS Watch and related projects in the coming months, when most of my time will be devoted to the JACSoN project with JISC Advance. We will develop and pilot a collaboration and communication portal based on Apache Rave and OpenSocial gadgets, building on the OpenConext middleware software that SURFnet in the Netherlands has developed to support federation across institutions. This is a very exciting project that will provide opportunities for researchers to use collaboration tools in groups across institutions, while using the credentials of their home institutions. I will blog more about that in the coming weeks.

I would like to take this opportunity to thank the university of Oxford and specifically the IT Services department, formally known as OUCS, for being such a wonderful host. I loved my time at the office and while I will continue coming in now and then, I will miss not being there every day. I also want to wish Scott and Mark a great time at OSS Watch and I’m confident that there is a excellent future for the service ahead with this refreshed team.

Governance of your open source project matters

When you are making your software open source, there comes a point when you have to think about how to respond to anybody that wants to engage with you or your team. As we have said on this blog before, a governance model is crucial in order to manage expectations and make sure you retain control where you need to, while sharing responsibility where you can.

We have published a set of documents about open source project governance models back in 2010, and have made sure they are kept up-to-date since. We also provide templates for a model with a Benevolent Dictator and one for meritocratic projects. All of our documents are released under a Creative Commons licence, which makes it easier to take our template and apply it to your own project.

If you set up a governance model for your project, people who have in interest in your project can easily see how your project is structured. It will help them understand how they can engage with the project, how decisions are made within the project, and how they may have an influence in the project’s strategic direction. If you are reluctant to give up influence in your project, you can design a model that will allow you to more tightly remain in control. When community development goals are more important, and you are looking to expand your community with others that may help guide the project in the right direction, the design of your governance model can reflect that. Whatever your approach, by making sure it is written down and the processes are thought through, you manage expectations of newcomers and prove to be a reliable partner for parties interested in investing in the project.

If you would like to see some real-world examples of governance models, I’m pleased to be able to say that there are a few respectable projects that have taken up our resources as an inspiration for their own governance model. The Qt project, for example, is governed by a meritocratic model that is based on our briefing note. Similarly, Yahoo!’s Mojito project uses a model that’s based on our meritocratic governance model.

While it may take some time to think through how you want to set up governance in your project, it is time well-spent. If you need any help with this, feel free to contact OSS Watch any time.

SURFnet as an inspiration for open innovation in HE collaboration

SURFnet in the Netherlands is roughly equivalent to JISC in terms of the role they play in the HE sector. They are doing some great work that is relevant to our support model and there is a great demonstration of open innovation at SURFnet that inspires JISC for the new ideas and services.

The Rave project has been mentioned earlier on this blog as an excellent example of open innovation. SURFnet was one of the initiating partners of Rave and donated the code of their local OpenSocial portal to the Rave project. However, if you follow the Rave activity closely, on the mailing list for example, you won’t see many people from SURF present there or contributing to that project. Does that mean they have simply dumped their code and moved on? Yes, but no! And that’s an interesting example of the enabling role SURFnet is playing in driving innovation in the HE sector in the Netherlands.

First some context. SURFnet provides many services to their HE community, centred around:

  • Federation services to enable single sign-on
  • Group management services that allow institutions to create and manage groups within and across institutions
  • Standards-based integration of services using the OpenSocial standard
  • Cloud-based services to enable institutions to collaborate

The platform that you will use to access all of these services is not centrally provided by SURFnet. That doesn’t mean that they do not support institutions that are looking at ways of integrating the tools for their users. If an institution wishes to build a portal on open source, SURFnet provides support to help them build it on the Apache Rave project.

All the code that SURFnet is developing for their services is released as open source with a permissive licence, often Apache 2. This enables other partners, such as JISC, to get started without the need to procure any specific software. Furthermore, it enables partners to contribute back and further improve the code for everyone. It would be great if we can build on that SURF model and collaborate with them to enable a similar best practice cross-institutional environment. And, even better, enable the creation and management of groups internationally! We are working on it, so stay tuned..

Government investment provides opportunities for open innovation

Getting more value out of our research investments is an important focus for the UK government. This became clear at the Science & Innovation conference last week, where several people from the UK department BIS and the Technology Strategy Board explained how new initiatives are being rolled out to increase uptake of research outputs by SMEs and other UK companies. We are watching this closely at OSS Watch, as this may provide additional opportunities for the software projects run at UK institutions for longer-term sustainability.

Kicking off the conference, Anne Glover (Chief Scientific Advisor for the EU Commission) talked about the focus areas of the EU’s major new investment programme, Horizon 2020. As much as 79.3 bn euro will be invested over the period 2014-2020, of which 24.4bn euro for Excellence Science and 17.9 for Industrial Leadership. But it will be crucial to help establish a good connection between these two areas, to make sure businesses benefit from research outputs, and there will be more focus on pushing the knowledge out of the universities, and involve industry more with the scientific projects.

The Rt Hon David Willetts MP, Minister for Universities and Science, reinforced this message in his talk, but on the UK level. He reminded the audience about the relative large share of SMEs in the British economy, compared with for example Germany, and indicated that more will be done help SMEs and universities connecting more easily. This is needed, because SMEs are finding it difficult to keep up-to-date with what is happening in universities and do not find their way easily to the right inventions. The HE Innovation Fund is intended help in this respect. Another example that David Willets mentioned is the Biomedical Catalyst Fund. This is a joined fund of £90m from both the Medical Research Council and the Technology Strategy Board, to help businesses take outputs of life sciences research and build innovations on top of those.

When looking at open source software, there is a great potential for institutions for those projects that are worthwhile preserving longer term, to allow external interested parties to contribute, including those from the commercial sector. It is important that the legal matters such as Intellectual Property ownership are sorted out properly, and that processes are in place to help businesses understand how they can invest and what they get in return, but once those ground rules are dealt with there are great opportunities for open innovation to thrive in these projects.

OSS Watch is here to help institutions make their projects ready for external involvement and make them more attractive for other partners. But we are also working to make it easier to find out which projects exist and may be of interest. For example, we are involved with a working group of the EU to help develop a metadata schema to make it easier for software forges to link out the meta data of their projects for consumption by other systems, such as our own software catalogue software Simal.

Improving the visibility and widening the opportunities for engagement with open source projects will be valuable for all. Open development has proven itself as a model for quite some years now, and now is a right time for projects to explore the potential for reaping the rewards of the opportunities open development brings. OSS Watch is here to help!

Jobs at OSS Watch – we’re looking for a community manager and a development manager

Do you have a passion for of free and open source software? Do you understand how open source communities work and do you enjoy working with people?

The University of Oxford hosts OSS Watch, an open source advisory service to the UK higher and Further Education sector. We assist in the use and development of open source solutions. We are currently looking for a Service/Community Manager and a Development Manager. An outline of the jobs are shown below, and the full description and instructions on how to apply are posted on the jobs website. Both posts are funded initially until 31st July 2013.

You can apply until noon (UK time) 2 July 2012, and interviews are scheduled for 12 and 13 July 2012. Feel free to get in touch if you have any questions.

For both roles, you will be a key part of an exciting project which is improving the methods of production of software in the UK academic sector. You should have relevant experience in an open and collaboratively developed open source project and must willing to undertake occasional travel to client sites and conference events.

Service and Community Manager

We are looking for a Service and Community Manager to lead a small team of specialists in delivering a national advisory service funded by JISC. For the community work, the main task is to encourage the sustainable adoption and development of open source software, where appropriate, in UK universities and colleges. The main focus of the role will be on community building to achieve longer-term sustainability for the projects and build links with wider open source communities. You will also lead the organisation of open innovation workshops and will present on community development and related open source topics at conferences and other events.

Development Manager

The Development Manager will lead our effort to actively support open development projects in UK institutions. We are looking for someone who will also actively educate the academic community about the more technical issues around free and open source software, helping projects proactively using the right tools and applying best practices of open development. OSS Watch encourages staff to spend an agreed amount of time on open source software development projects where they are compatible with the service’s remit.

Get started with research data management using open source applications

Researchers generate ever-increasing amounts of data when performing their research and they need to find new ways of managing this data properly. This process is accelerated by the research councils and other funders in the UK, who are increasingly requiring bids to indicate how they will manage the data that is part of the research that is to be funded. I attended the second day of the ‘roadshow’ organised by the Digital Curation Centre (DCC) in London last week, where Sarah Jones presented about the funders’ data policies. The DCC have an excellent concise overview of funders’ requirements, which shows that nearly all now expect policy stipulations regarding data management planning and sharing. Sarah showed that aspects that are important to nearly all funders are timely release of data, open data sharing wherever possible and provisions for longer-term preservation of the data.

The burden for much of this will be placed on the institutions, who need to have a clear data management policy in place, as well as the tools to support all the aspects of data management. The DCC is helping institutions with this, via the excellent resources on their websites, tools such as DMP Online to help them create data management plans, and by organising events such as the roadshow where experiences can be shared and best practices disseminated.

On the more technical level, there are a number of open source tools available that will allow departments and institutions to manage research data. HEFCE and JISC have funded a number of projects, that release their software as open source. A few examples are:

  • DataStage is a secure personalized ‘local’ file management environment for use at the research group level
  • DataBank is a scalable data repository designed for institutional deployment.
  • VIDaaS (Virtual Infrastructure with Database as a Service) is a project of two halves. The ‘DaaS’ part will develop software that enables people to build, edit, search, and share databases online; the ‘VI’ part involves the development of an infrastructure enabling the DaaS to function within a cloud computing environment.
  • BRISSkit will design a national shared service brokered by JANET to host, implement and deploy biomedical research database applications that support the management and integration of tissue samples with clinical data and electronic patient records.

There is an excellent opportunity for institutions and research department to start trialling these tools without the need to make large investments. And if the tool fits your use case, it is easy to get involved with the community and benefit from the opportunities that the open development approach offers. OSS Watch is here to help!

The dominance of open source tools in Big Data

Most of the tools that are best suited for dealing with Big Data are open source. This provides the research community with a huge opportunity, because no investment in software licenses is needed. You just download the software and ‘get on with it’. The challenge, as became clear at the Eduserv symposium last week, is to find people with the right skills to apply these tools.

Without a doubt, Apache Hadoop that is the most important open source project in this space. It is amazing to see how fast the Apache Hadoop ecosystem is growing and how everyone is trying to jump on the bandwagon. Start-up companies like Cloudera and Hortonworks have no trouble finding venture capitalists willing to invest large sums of money. Similarly, nearly every major tech company is offering it, while other internet companies that deal with big data are using it (secretly or not). At the Eduserv symposium, EMC CTO Rob Anderson focused on the implication big data has for storage, and showed their Hadoop-based offering. Because the Apache licence allows you to use any Apache project in a closed-source implementation, EMC can sell their Hadoop distribution without needing to make that product open source.

There are big implications of the big data trend for the research community. Guy Coates of the Sanger institute showed how the amount of data they are managing is increasing rapidly. They are expecting this increase to continue, especially since the costs of human DNA sequencing is dropping dramatically. They expect it to drop to $1000 for a full scan within two years (excluding storage!). His main challenge was not the actual storage of the data, but the management of the data as researchers were analysing it. Sanger is using the open source tool iRODS, a community-driven project that originates from the Data Intensive Cyber Environments (DICE) research group in the DICE Center at the University of North Carolina.

Another open source project that featured prominently at the Eduserv symposium was Apache CouchDB. Simon Metson of Bristol University explained how NoSQL is the enabler of big data and new database systems that do not use the traditional relational database approach are better suited for these tasks. Open source software projects like CouchDB, but also Apache Cassandra, are leading in this space. Simon highlighted that the community-aspect of big data is very important. By engaging with the community that uses these tools to solve their big data problem, you can solve the hard problems. Something you may encountered once in a thousand times, may have been solved by someone else in the community who runs into it more often, and vice versa.

The closing keynote was given by Anthony D Joseph, professor at the AMP Lab at UC Berkeley. He mentioned how Facebook started the Open Compute project to share best practice in cluster design for big data centers. It is an interesting example of the old economic adage that you should commodotise your complement. Berkeley is collaborating on the Apache Incubator project Mesos, which is a scalable cluster manager that can dynamically share resources between multiple computing frameworks. They support frameworks like Hadoop, Spark and MPI.

So the technology is there or is well underway in being developed. And being open source, anyone can download and start using it. Technology is not the problem of big data, but the challenges lie in the cultural and organisational change that is needed to capitalise on big data. People within and across the organisations need to be willing to share their data and think of new, intelligent and creative ways of making use of this data. Two well-known examples that were mentioned were the Google flutrends, a website that predicts flu epidemics based on what people search for, and a Twitter application that was created to detect and report on earthquakes using people’s tweets.

A final challenge that was recognised widely at the conference was the shortage of skilled people in the big data space. This is true both for the data scientists that were needed to analyse the data, and for people that can help curate the data longer term, which is a completely different challenge for many HE institutions. In the spirit of open source though, there are many resources freely available online for people who want to get started, such as on the website bigdatauniversity.com. And of course, if you want to get started with one of the open source projects mentioned, there are many ways to get involved.

Don’t keep your data under your desk

It is a well-known problem for researchers. Data is being collected for a research project and no decision has been made about how to manage the data during the project. Naturally, once you have finalised the project and start publishing on the end results, you may deposit your final dataset in a institutional repository such as your university’s DSpace or E-prints repository, or you may even put it in Dryad. However, that is not sufficient to keep your data safe while you are still working on it. Often, such data ends up on a computer that just happens to lie around in the office or department, or even on the researcher’s local machine.
Continue reading

Graduating Apache Rave project demonstrates open innovation in software

The Apache Rave project graduated from the Incubator last month. This means that the Rave project has demonstrated to be a viable project community, which is being governed well according to the meritocratic principles of the Apache Software Foundation.

Apache Rave provides a next-generation portal engine, supporting (Open)Social Gadgets as well as WC3 widgets. Have a go with the latest release and you will see that it works out-of-the-box, but it can alternatively serve as the basis for an enterprise-level social portal application.

Continue reading