Using AI to analyse collections

Jonathan Knott, 14.02.2017
AI techniques could help curators do their jobs
Many cultural institutions are now in the process of digitising their collections and making them available online. But their holdings are often so vast that it can be hard to know how to manage, present and interpret them, and some museums are turning to artificial intelligence (AI) to help them do this.

Mia Ridge, the digital curator at the British Library and chair of the Museums Computer Group, says it is important for the cultural sector to understand AI because it is increasingly being used to manage collections. “I think it will help us enhance records and categorise our collections,” Ridge says.

The British Library is experimenting  with machine learning in a number of different ways. It is trialling software called Transkribus that uses machine learning to read handwritten text using what it has learned from analysing existing examples of transcribed text.

While there is a long way to go in terms of accuracy, automated text transcription has huge potential, Ridge says. It would make collections easier to search and automatically find references to entities such as people, places, dates and events.
In another experiment run in collaboration with University College London, the British Library used machine learning to help categorise and tag digitised images from 16th- to 19th-century books. The results can be seen at the British Library Machine Learning Experiment site.

Mario Klingemann, an artist who works with algorithms and data, has also used machine learning to work on the same digital collection. In 2014, he created a system that added category tags to about 500,000 of the images, which had been uploaded to Flickr.

“The high resolution images were beautiful material to work with. But the problem was that it was all unlabelled, so you couldn’t search for something you were looking for,” he says.

“You have to look at a lot of images manually first,” he says. “I started looking at them, and splitting them into subcategories. At the same time, I was training the model and figuring out ways that it could distinguish a portrait from a landscape image, for example. As the model got better, I had less and less manual work.”

Following this project, Klingemann became an artist-in-residence at the Google Cultural Institute in Paris, where he created the X Degrees of Separation project with support from Google engineers. This allows visitors to select two images from the digitised collections held on the Google Arts and Culture site, and then creates a pathway between them – a series of intermediary images that form a gradual visual transition.

“What I really like about it is that it creates this serendipity,” says Klingemann. “You might have started with two artefacts that you know. But on the way between them you will get to see a lot of objects that you might have never seen before, or you didn’t expect.”

The technology that the project is based on uses deep learning – a type of machine learning that can learn more complex patterns in data.

X Degrees of Separation is one of several experiments hosted on a Google Cultural Institute microsite that provide playful ways to explore the digitised partner collections it hosts, using 3-D animations and thematic links.

These projects are not designed with any specific purpose in mind. But some research has more direct relevance to the work of art historians and curators. Researchers at Rutgers University (New Jersey), for example, have used machine learning to identify influences between paintings, and even claim that the technology has spotted a link not made before by art historians.
John Davies, a research fellow at the innovation charity Nesta who focuses on the creative and digital economy, says that the capacity of machine learning techniques to recognise patterns could potentially be put to use for tasks like analysing similarities between paintings and spotting forgeries.
“Humans can spot similarities between pictures and a computer is able to do that as well. But because the computer can process many more pictures and remember them, it can analyse a very large number at one time,” he says. “In principle, a computer could be trained to recognise works by a certain artist.”

Algorithms that are able to recognise images and faces are well established. A project in The Netherlands created ‘The Next Rembrandt’ by digitally analysing the artist’s work to learn his style.
Brendan Ciecko, the chief executive of the digital platform Cuseum, believes AI techniques could play a significant role in helping curators do their jobs in future.

“There are now bots and scripts that are able to create paintings, and write music and screenplays,” says Ciecko.

“So the possibility of AI writing object labels, assisting with interpretation and scripting audio guides is completely feasible.”