AI scraper bots are disrupting online collections, report finds

A growing number of cultural heritage institutions are reporting disruption to their online infrastructure caused by AI scraper bots, according to a new report by the Glam-E Lab.

The lab, which is a joint research initiative between the Centre for Science, Culture and the Law at the University of Exeter and the Engelberg Center on Innovation Law & Policy at NYU Law, circulated a short survey earlier this year asking institutions about their experiences with bots building AI training datasets.

The research particularly focused on digitised collections and data connected to galleries, libraries, archives, and museums (Glam).

The survey came in response to isolated reports of bots swarming servers and online collections, in some cases overwhelming their systems and knocking them offline.

The research found that reports of bots impacting online collection are not isolated experiences and that such behaviour is “widespread and likely growing”.

“There is no single solution to the problems it is creating,” the report said.

The survey found that bots are widespread but not universal, with 39 out of 43 respondents saying they had experienced a recent increase in traffic.

Twenty-seven of those 39 respondents attributed the increase in whole or in part to AI training data bots, with an additional seven believing that it might be attributable to bots.

The increased bot activity is difficult to measure because few respondents were actively tracking bot traffic until it triggered a crisis in their collection, according to the research.

“Many respondents did not realise they were experiencing a growth in bot traffic until the traffic reached the point where it overwhelmed the service and knocked online collections offline,” the report said.

The report found that some respondents have been seeing an increase in bot traffic since 2021, while others did not experience their first spike until 2025.

Some of the training bots clearly identify themselves, while others take a range of measures to hide their source.

Bots tend to swarm for relatively brief periods of time, the survey found, but the frequency of swarms may be increasing, according to the research.

The report found that while the impact of bots can be uneven, “one constant does seem to be that everyone notices when a site goes down, or slows to a crawl, because it is being swarmed by an unsustainably large collection of bots”.

Solutions

Cultural heritage institutions are using a range of home-grown and third-party firewall solutions to try to screen out the bots. “Some of these efforts appear to be effective, although few are confident that they will be sustainable in the long term,” said the report.

Some respondents reported having limited success in reporting problematic bots.

Respondents indicated that they were reluctant to take “more aggressive” steps to move collections behind, for example, login screens.

This was for a variety of reasons, including doubts about how effective those measures would be in the medium term, as well as concerns that implementing such changes could have negative impacts on welcoming users and run counter to the institutions’ larger goal of making their collections easily available online.

Respondents said they were concerned that bot swarms would “create an environment of unsustainably escalating costs for providing online access to collections”.

In addition to disrupting access to collections, the report found that bot swarms affected website analytics, making it difficult for institutions to get an accurate understanding of user growth. “Multiple respondents described a dawning realisation that traffic growth that had been attributed to an increase in human visitors was, in fact, simply the early signs of a bot swarm,” said the report.

The report said that online collections are targeted because they are seen as “high-quality sources for data to train models”.

“Machine learning and artificial intelligence models… are trained on large quantities of data,” it said.

“These collections are often easily accessible and well structured for machine readability... As such, teams building AI models have drawn on them for some time, regardless of whether they are openly licensed or subject to copyright restrictions.”

The report expressed optimism that a “workable balance” could be found with bot creators.

“The cultural institutions that host online collections are not resourced to continue adding more servers, deploying more sophisticated firewalls, and hiring more operations engineers in perpetuity,” the report said.

“That means it is in the long-term interest of the entities swarming them with bots to find a sustainable way to access the data they are so hungry for.”

The report concluded: “Perhaps the best hope for the future of online access is that AI training dataset bots fade into the background noise of the internet. They add traffic, but at manageable levels... And maybe they will help more people discover these collections, build on them, and create something new.”