Agencies run funding programmes with an expectation that many will not lead to sustainable innovations in the longer term. But is the structure of funded projects itself causing projects to fail?
Dr Cathy Gunn and Rhiannon Herrick have been investigating this issue, looking at 22 projects in Australasian higher education from the past five years, resulting in the Sustaining eLearning Innovations report published by ACODE.
Their conclusions are that, while “its unreasonable to expect every great idea to translate into a sustainable system”, that “institutional and national strategic intentions are not reflected in what is happening at practice level”.
In the study, of the 22 projects they looked at, 16 did not meet their criteria for sustainability:
This is largely due to continued dependence on the initial innovator for support for everything from answering calls from users to sourcing additional funding for further development and maintenance
Sustainability of open source software developed by funded projects is also an issue we’re concerned with here at OSS Watch so I met up with Dr. Gunn at the ALT-C conference in Manchester, where she presented some of her findings.
In her talk, Dr. Gunn identifies six key problem areas for funded projects:
- Scoping
- Funding constraints
- Impact and influence
- Planning for the future
- Owning the innovation
- Skills for different stages
1. Scoping
According to Dr. Gunn:
Most projects are started by an individual with an idea and a passion to translate the concept into reality. Many began because there was ‘nothing available to meet their requirements’ but the scoping process used to reach this conclusion is not clear and there is no evidence that information gathered at this stage is used to inform the development.
[..] it appears that innovators have an idea about what they perceive to be an educational problem and exactly how it should be solved. If they cannot find something that solves their problem in their way, the conclusion is ‘there’s nothing like it’ and they go ahead and create it.
One of the common lines of defence for creating new software without any clear path to sustainability is that ‘nothing met the requirements’. This is something that I’ve heard a great deal when projects present their work and Dr. Gunn is right to point out that this is rarely backed by evidence that the project made a serious effort to find existing work to adapt or build upon.
At a project level I think this can be addressed in two ways. One is by conducting a serious effort to identify and work with existing software – for example, by identifying existing open source communities to engage with to add new capabilities. Another way is to shift the emphasis from development to active use, for example, to scope the project to fit the capabilities of existing software.
The fact remains however that quite a few of the individuals that engage in innovation are, by nature, passionate about and motivated by the idea of developing new systems. However, that passion needs in some cases to be diverted into improving the sustainability of existing software rather than “reinventing the wheel”.
The Sustaining eLearning Innovations report recommends that “Innovators and funding bodies use available evidence, or identify experts or colleagues who are able to assist with more accurate scoping of projects”. This would mean, for example, that funders should exercise more diligence in identifying whether projects were replicating other work, or planning to develop software that already existed, or could be modified to suit their requirements.
In all fairness, funding agencies do try to do this already using external reviewers; however it tends to rely on collective memory of previous projects and programmes rather than a search of existing solutions. Perhaps funders ought to consider such a search as an additional part of the review process, or include reviewers on panels whose remit is to perform a technology search?
The alternative is to rely on projects to perform such diligence at the proposal stage; this is something common in European funding regimes which require a “state of the art” and “beyond the state of the art” component in proposals. The problem lies then in the capacity of reviewers to sufficiently critique the scoping.
2. Funding constraints
Dr. Gunn raised the issue of funding periods; asserting that:
Broad consensus is that project funding for two or three years is sufficient to produce a full working prototype, but not a finished product that is disseminated and sustainable. Externally set parameters like timeframe and budget don’t necessarily serve innovations well. It’s often hard to anticipate what will be achieved within a 2 or 3 year period.
This is a continuing issue for funding agencies. In some cases, programmes provide second-round funding. In the UK, JISC created a series of “Benefits Realisation” grants that were aimed at supporting sustainability activity – such as packaging project outputs and providing training materials. However, in most cases, and especially for larger projects, the structure is such that all funded activity, including sustainability work, has to be conducted within the project timeframe.
This means that projects need to focus on tackling sustainability as early as possible – from pre-proposal stage onwards – rather than tacked on at the end, naively assuming that the design and implementation work will complete on time leaving some time at the end to figure out the end game.
This is why at OSS Watch, we’re keen to engage with projects early on in the process, so that sustainability activities can be planned and executed earlier in the project timeframe.
An alternative model also applied at JISC has been to do much smaller “rapid innovation” projects – for example six month projects with relatively small amounts of funding. This can operate as an “incubator” for identifying potential concepts worth a second funding round, or for further development by the originating institution using their own resources. Paradoxically, the answer for having too few resources allocated to dissemination and sustainability may be to cut funding for it altogether, at least in the initial development stage.
The opposite approach has been to create slow-burn projects that are funded over a longer timeframe – four or five years. The risk here though is that, while there is sufficient time to develop a sustainable product, its hard to predict whether its going to be worth it as so much can change during the project lifetime.
In general, what I’ve observed is that projects and institutions favour larger projects as it makes resource planning easier, while funding agencies often see better results from shorter, smaller projects, but this requires more support, coordination, and – critically – follow up activity.
3. Impact and influence
It may seem surprising given how much emphasis is placed on “impact” these days to find that projects are failing to communicate the benefits of their work; however Dr. Gunn points out that the evidence of benefits that projects provide may be “not in a form that influences players, such as IT departments and those making management level decisions about use of a product within an institution”.
This mismatch between the perspectives of innovators and decision makers has been a major topic in business generally, but is acutely felt in higher education. There have been some interesting initiatives by funders to address this issue – for example some JISC programmes have included specific benefits realisation activities aimed at developing a more coherent message about benefits for potential adopters of the innovation. Others have used taken a user-focussed approach (e.g. the Users and Innovation programme) with a model that encouraged projects to communicate in a less project-centric fashion. I’m not aware however of any analysis of the success of these approaches; anecdotally (being involved in some projects in these programmes) it did seem to have an effect.
4. Planning for the future
Many funded projects do not have a clear roadmap for future development, support, dissemination or sustainability. Dr. Gunn noted that “The main focus of projects is on design, development, implementation and impact evaluation. A different set of skills is required for dissemination, support and ensuring sustainability.” Also, even where both sets of skills are present in a project team or consortium, there can be issues with communications between the groups.
From my own experience, even where programmes require the development of an exploitation plan including sustainability, this is sometimes either done in a very half-hearted way, or is wildly optimistic about the impact that can be achieved with the resources and effort applied.
For example, I still routinely see proposals for making successful open source software that begin with selecting a license, and end with dumping the code on SourceForge hoping a community will magically appear.
Developing realistic and effective strategies for open source community engagement is something OSS Watch has a lot of expertise in, so I’d recommend reading the articles on our site if you’re responsible for such a plan, or to get in touch with us.
5. Owning the innovation
Dr. Gunn puts this very succinctly:
Universities often fund development and are seen by the innovators to ‘own’ the products, but this ownership typically does not translate into meaningful actions to sustain them.
This is something I’ve also heard described as the “Fred in Shed” model – that innovation projects are started in institutions at a safe remove from core services, metaphorically in the shed at the end of the garden. However there is often either no process for moving the innovation into the centre, or for investing in innovations that come from within the host institution. Some are certainly better than others – for example, the Rogō project that was the subject of an earlier post was adopted by central services using its own procurement process – effectively treating the internal project in the same way as an external supplier bidding for the solution being adopted.
In some cases however there seems to be an inversion of “Not invented here”; or as Dr. Gunn puts it:
The bottom line is that innovators and their work are not valued as highly as they should be by colleagues and employers.
Even worse, the host institution is often seen as a barrier, rather than a potential owner:
Around half of the case studies reported deliberate moves to subvert IT Services controls, which they believe would have killed their ideas in the early stages.
Part of the answer may be to try to meet half-way – for central services to smooth the path for innovators and projects so they don’t try to subvert the controls they need to put in place, and for innovators to take the time to understand the concerns of central service departments and work with them rather than against them. At least then there is potential route from innovation to service.
However, the larger strategic issue is whether higher education institutions are willing and able to invest in developing innovation beyond externally-funded projects.
6. Skills for different stages
As also noted under “Planning for the future”, the skills needed for design and implementation aren’t the same as those needed for dissemination, sustainability and community building and as Dr. Gunn notes, its unrealistic to expect the innovator and development team to provide them all.
What can projects do?
Many of the proposals in the report are aimed at funding agencies rather than projects, however there are clearly actions that can be taken by projects at the proposal and pre-proposal stage to mitigate some of these problems.
In the UK, projects can engage early on with innovation support services such as OSS Watch, CETIS and UKOLN to help with the scoping phase; identifying how their proposal can have a better chance of sustainability by building on earlier work, participating in existing open source communities, or adopting existing standards.
Projects can also talk to the potential owners of the innovation in their institution much earlier – for example, meeting with colleagues in the service departments of their institution before proposals are fully developed to explore whether, if successful, the innovation could be adopted and further developed locally as part of the services portfolio. This may provide proposers with more realistic constraints to work with – for example, to only build on platforms that can be locally deployed and supported. This is something that has always been a bit of a problem for innovators – and a reason why as the report notes projects often circumvent central services – but I think if projects want to have a serious shot at success then they need to think of ways they can work with rather than against their local service units. It may make sense to develop prototypes outside of local constraints – but if done without any understanding of the local services context it can be a missed opportunity for getting in-house support and investment later on.
Project teams can also be more diverse, and include colleagues or partners with skills in sustainability, community building and dissemination. However, they need to be engaged from the start, not brought in at the end of the project.
In conclusion, I think anyone who has worked on funded projects will recognise the problems that Dr. Gunn has pointed out in her work; however I think given the more challenging funding climate its time we all took them much more seriously.
You can download the Sustaining eLearning Innovations report from the ACODE website (PDF, 24 pages). If you’re developing a proposal and are based in a UK college or university, you can contact OSS Watch for advice by emailing info@oss-watch.ac.uk.
