Sector warned of ‘acute threat’ posed by AI scraping bots

AI scraping poses substantial and unsustainable costs on cultural heritage institutions, which are “existential to the mission of these bodies”.

That’s the warning from a new white paper for policymakers based on research conducted by the University of Edinburgh in 2026 as part of the Ekip (European Cultural and Creative Sectors and Industries Policy Platform) project.

Since 2022, there has been a massive increase in automated bot traffic from AI companies scraping data to train generative AI models. Re:Commons: Renewing the Digital Commons for Cultural Heritage found that these bots are putting a strain on organisations’ digital infrastructure, which in some cases is comparable to cyberattacks.

Dealing with over-burdened servers has a knock-on effect on engineering and cloud-based hosting costs, as well as staff members’ time and energy.

“For institutions with shrinking public budgets, these costs divert resources from their core mission,” the report warns. “Paradoxically, institutions are also experiencing a loss of legitimate web traffic [due to AI tools like ChatGPT].”

The negative impact on staff morale is also a risk, which stakeholders in the study describing bot traffic as “overwhelming” and “intense”.

Other risks highlighted in the study include copyright infringement, a general distrust of commercial AI models and a loss of context, provenance and control.

Policy measures

Re:Commons warns that current policy debates about AI scraping bots, dominated by copyright and licensing questions, risk overlooking “the infrastructural and material dimensions of the crisis and the specific vulnerability of publicly funded, commons-based institutions”.

The research, undertaken with 115 cultural heritage stakeholders, found that while there is a willingness to share data with AI companies, the combination of unsustainable traffic and lack of dialogue with the companies using the bots has led to reactive measures such as bot-blocking firewalls. 

“These defensive responses are quietly creating new enclosures that undermine the very openness that cultural heritage institutions seek to protect,” the report warns.

The paper offers recommendations that aim to not only mitigate the costs of AI bots but also “reimagine AI and the commons on more equitable terms”.

These include:

• Developing and adopting machine-readable rights-reservation standards.
• Requiring AI companies to offer routes to recourse for unlawfully scraped content and to uphold data protection and right to erasure
• Mandating bot identification, redress and transparency pathways.
• Establishing compensation mechanisms that pay for infrastructure rather than merely licensing content.
• Investing in shared, sovereign, environmentally sustainable public digital infrastructure.
• Supporting small-scale, community-governed and public-interest AI models aligned with the values of cultural heritage.

What are museums doing to mitigate the impact of AI bots?

According to the research undertaken by the University of Edinburgh, organisations have been forced to “act alone” to deploy some of the following defensive measures to protect their infrastructure:

1. Server upgrades “This is a reactive strategy that treats symptoms rather than causes, available only to institutions with the budgets to sustain it,” the report’s authors note.
2. IP blocking and rate limiting “These measures are blunt and error-prone,” the report states.
3. Third-party firewall services “In defending themselves against one form of private extraction, institutions become dependent on another set of intermediaries that now sit between the public and public cultural heritage,” the authors say.
4. Whack-a-mole and the blocking paradox The research found mitigation strategies shift as bots learn to get around the above measures but warns: “Blocking also often fails to achieve its aim.”

The reports authors are: Caitlin McDonald, Morgan Currie, Lara Dal Molin, Nicola Osborne, Anna Talley, Matea Senkić, and Jennifer Volk.

The research team has contributed to two European Commission policy calls for evidence, on the impact of AI scraping bots on the gallery, library, archives and museum (GLAM) sector and on copyright environment for cultural heritage institutions in relation to scraping by AI companies.