Announced 5 Nov 2026 · Sources checked
What is happening to university libraries?
University libraries across North America are experiencing unprecedented surges in automated traffic from AI web scraping bots, threatening to degrade or block public access to vital digital resources. Library administrators report traffic increases exceeding 1000% of normal levels as AI companies aggressively harvest academic content to train large language models.
The bots target libraries' digital repositories, academic databases, and digitized collections of historical and government documents that are not behind paywalls. These open-access archives, built over decades with public funding, have become prime targets for AI companies seeking training data that is free, high-quality, and legally ambiguous to scrape.
Jill Strykowski, a cataloging and government documents lead at San Jose State University, described bot behavior as bombarding, saying they try to grab up as much material as possible. She called the bots a headache for institutions that provide open access to digitized work.
How bad is the traffic surge?
The numbers paint a stark picture. Marmot, a nonprofit consortium of academic, public, and school libraries across Colorado, told the Littleton Independent that bots slow service and occasionally cause outages. Earlier in 2026, the system was seeing around 6 million bot connection attempts every 24 hours, totaling 180 million attempts in 30 days. That number has now skyrocketed to over 2 billion attempts in a 30-day period.
At the University of North Carolina at Chapel Hill, librarians recorded over 11,000 searches for Finnish music from thousands of distinct IP addresses in a single day. The pattern is consistent with AI scraping bots that request thousands of documents per second, unlike compliant crawlers that follow robots.txt directives. The bots employ sophisticated techniques to mimic human browsing patterns, cycling through proxy servers and residential IP addresses to evade basic blocking measures.
How do bots break into restricted collections?
Beyond scraping public materials, some bots actively work to breach restricted collections. Rachel Fewell, a director at Englewood Public Library, described how bots test the system to see if they can access a working library card number. If successful, bots max out everything they can check out and scrape the content.
This account fraud dimension adds a security layer to what was initially a capacity problem. Libraries that offer digital borrowing services find their patron authentication systems under constant probing. IT teams describe a continuous escalation cycle: they implement countermeasures, and bot operators adapt with new evasion tactics within hours.
What about physical books being destroyed?
The data hunger extends beyond digital archives into the physical world, with more permanent consequences. In a notable investigation, 404 Media journalists slipped an AirTag into a shipment of rare books and tracked it to an Amazon AI training facility in Las Vegas, Nevada, where books are reportedly torn apart before scanning.
Anthropic has reportedly cut millions of books from their bindings for easier scanning and discarded them after making digital copies. The Kyobo Book Centre in South Korea recently turned away a 100,000-title order it suspected was destined for AI training, highlighting how the demand for training data now drives acquisition of physical books at industrial scale.
Unlike digital scraping, which temporarily slows systems, physical destruction is permanent. Rare and out-of-print books, once destroyed, cannot be replaced. Cultural heritage institutions are watching this trend with alarm.
How are libraries fighting back?
Institutions are implementing multi-layered defense strategies. Technical measures include advanced bot detection systems using behavioral analysis and machine learning, dynamic rate limiting that adapts to traffic patterns in real time, IP reputation scoring that prioritizes traffic from known academic networks, and CAPTCHA challenges for suspicious access patterns.
Some libraries are updating their robots.txt files and terms of service to explicitly prohibit bulk automated scraping. Earlier this month, OpenAI, Google, Microsoft, and Anthropic backed a shared robots.txt signal to let sites block AI training in one step, but the signal remains voluntary and libraries report that many scrapers ignore it entirely.
Smaller university libraries face particularly acute challenges. Unlike major research institutions with robust infrastructure budgets, regional universities lack the server capacity and cybersecurity resources to absorb massive bot traffic while maintaining service quality. Several institutions have been forced to implement aggressive rate limiting that inadvertently impacts legitimate users.
What is at stake for open access?
The crisis highlights fundamental tensions between academic values of open access and the practical need to protect infrastructure. Universities have historically championed unrestricted access to research and knowledge, making many administrators reluctant to implement aggressive blocking that might inadvertently restrict legitimate use.
Library associations are calling for AI companies to voluntarily adopt ethical scraping practices, including respecting robots.txt directives, limiting request rates, and identifying their bots clearly. However, the competitive dynamics of AI development create strong incentives for aggressive data acquisition regardless of impact on public infrastructure.
The resolution will likely require a combination of technical defenses, policy frameworks, and industry cooperation. Without systemic change, the open access principles that underpin academic research worldwide could be eroded by the very technology they helped train.
Common questions
Are these bots breaking the law?
Most bots are not breaking specific laws, as the content they scrape is publicly accessible. However, bypassing rate limits, stealing library card credentials, or ignoring terms of service may constitute unauthorized access or computer fraud in some jurisdictions.
Can libraries simply block all bots?
Not easily. Blocking all automated traffic would also block legitimate research tools, search engine indexing, and accessibility services. Libraries must balance protection with their mission of providing open access.
Is this only a US and Canada problem?
No. The trend is global. University libraries in the UK, Australia, and Europe have reported similar surges in bot traffic as AI companies expand their training data acquisition efforts worldwide.
Sources & further reading
How this story was made
Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. The cover is an AI-generated editorial illustration, not a photograph or evidence of an actual event.
Our editorial standards
