The Wikipedia hoover

Six frogs from a paper available on PubMed

“Pick a frog, any frog” – an image automatically imported from PubMed to Wikipedia

Back in August Wikimania came to London and I heard some interesting discussion there of Wikipedia’s approach to open access materials and the tools they are developing to support that approach. This github repo contains some interesting open source projects designed mainly to automate the process of identifying cited external resources that can be copied into Wikipedia’s repositories of supporting material wikisource (for texts) and (for pictures, video and sound).

open-access-media-importer for example is a tool which searches the online repository of academic biology papers PubMed for media files licensed under the Creative Commons attribution licence and copies them into the wikimedia repository. Where the files are in media formats that are encumbered by patents, the script also attempts to convert them to the patent free ogg format framework.

In the same github repo is the OA-Signalling project presents a developing framework for flagging open access academic papers using standardised metadata, perhaps integrated in future with the systems being developed by DOAJ and CrossRef. This wikipedia project page explains further:

Some automated tools which work with open access articles are already created. They impose nothing upon anyone who does not wish to use them. For those who wish to use them, they would automate some parts of the citation process and make an odd Wikipedia-specific citation which, contrary to academic tradition, notes whether a work is free to read rather than subscription only. The tools also rip everything usable out of open access works, including the text of the article, pictures or media used, and some metadata, then places this content in multiple Wikimedia projects including Wikimedia Commons, Wikisource, and Wikidata, as well as generating the citation on Wikipedia.


During the sessions in which open access and these tools were discussed, many participants expressed strong dislike for academic publishers and their current closed practices. Clearly for many the idea that Wikipedia could become the de facto platform for academic publication was a charming idea, and more open access was seen as the best route to achieving this.

Many years ago I worked in a digital archive, and one of the problems we faced was that academics who were depositing their databases and papers wanted to be able to revise them and effectively remove the earlier, unrevised versions. Naturally this made our jobs more challenging, and to a certain extent seemed to be opposed to the preservation role of the archive. My experiences there make me wonder how the same academics would react to their papers being hoovered up by Wikipedia, potentially to become unalterable ‘source’ copies attached to articles in the world’s most used reference work. On the one hand it is a great practical application of the freedoms that this particular kind of open access provides. On the other hand, it perhaps risks scaring authors into more conservative forms of open access publication in the future. Personally I hope that academics will engage with the tools and communities that Wikipedia provides, and handle any potential friction through communication and personal engagement. And in the end, as these tools are open source, they could always build their own hoover.