Robots.txt and Crawler Politeness in the Age of Generative AI
PDF Version Also Available for Download.
Description
Poster providing an overview of the robot.txt standard including results from a study of its current use and potential uses for generative artificial intelligence (AI). It was presented at the IIPC General Assembly and Web Archiving Conference held on April 8-10, 2025 in Oslo, Norway.
The mission of the IIPC is to acquire, preserve and make accessible knowledge and information from the Internet for future generations everywhere, promoting global exchange and international relations.
Descriptive information to help identify this poster.
Follow the links below to find similar items on the Digital Library.
Description
Poster providing an overview of the robot.txt standard including results from a study of its current use and potential uses for generative artificial intelligence (AI). It was presented at the IIPC General Assembly and Web Archiving Conference held on April 8-10, 2025 in Oslo, Norway.
This poster is part of the following collection of related materials.
International Internet Preservation Consortium (IIPC) General Assembly and Web Archiving Conference
Presentations, abstracts, posters and other materials from the International Internet Preservation Consortium's (IIPC) annual General Assembly and Web Archiving Conference.
Nagel, Sebastian & Vaughan, Thom.Robots.txt and Crawler Politeness in the Age of Generative AI,
poster,
April 9, 2025;
(https://digital.library.unt.edu/ark:/67531/metadc2472442/:
accessed June 15, 2026),
University of North Texas Libraries, UNT Digital Library, https://digital.library.unt.edu;
crediting International Internet Preservation Consortium.
