n.an open source web crawlerProm and Swain 2007, 360Heavy-duty webcrawlers, such as the open source Heritrix, can be used to download large amounts of data. However, Heritrix is very difficult for all but the most tech-savvy archivists to install and use.Milligan 2019, 119While modern versions of the standard Heritrix web crawler—Heritrix being the program that goes throughout the web and systematically takes snapshots of pages—will now access this material, older versions did not.Wickner 2019, 5The Internet Archive maintains Heritrix, an open-source web crawler. Archive-It, a popular Internet Archive subscription service for managing institutional web archives and archiving, incorporates Heritrix among other technologies to perform captures.Lohndorf 2022Heritrix is the Internet Archive’s open-source, extensible, web-scale, archival-quality web crawler and has been widely used by many different organizations for nearly 2 decades.
