Politely Downloading Millions of WARC Files Without Burning the Servers Down
PDF Version Also Available for Download.
Description
Poster sharing the development and release of an open source, cross-platform, dependency-free and user-friendly tool that implements an HTTP client with an easy to configure retry-strategy with exponential backoff and jitter. The introduction of these more polite retry strategies allow users to avoid download errors, and to more quickly download data as the exponential backoff with jitter lets the server more easily handle concurrent requests. It was presented at the IIPC General Assembly and Web Archiving Conference held on April 8-10, 2025 in Oslo, Norway.
The mission of the IIPC is to acquire, preserve and make accessible knowledge and information from the Internet for future generations everywhere, promoting global exchange and international relations.
Descriptive information to help identify this poster.
Follow the links below to find similar items on the Digital Library.
Description
Poster sharing the development and release of an open source, cross-platform, dependency-free and user-friendly tool that implements an HTTP client with an easy to configure retry-strategy with exponential backoff and jitter. The introduction of these more polite retry strategies allow users to avoid download errors, and to more quickly download data as the exponential backoff with jitter lets the server more easily handle concurrent requests. It was presented at the IIPC General Assembly and Web Archiving Conference held on April 8-10, 2025 in Oslo, Norway.
This poster is part of the following collection of related materials.
International Internet Preservation Consortium (IIPC) General Assembly and Web Archiving Conference
Presentations, abstracts, posters and other materials from the International Internet Preservation Consortium's (IIPC) annual General Assembly and Web Archiving Conference.
Ortiz Suarez, Pedro; Vaughan, Thom & Lindahl, Greg.Politely Downloading Millions of WARC Files Without Burning the Servers Down,
poster,
April 9, 2025;
(https://digital.library.unt.edu/ark:/67531/metadc2472441/:
accessed June 7, 2026),
University of North Texas Libraries, UNT Digital Library, https://digital.library.unt.edu;
crediting International Internet Preservation Consortium.
