n.synonym for web crawlerProm and Swain 2007, 360The filters in HTTrack are powerful enough so that we could direct the crawler to use the RSO list to harvest student organization sites from distributed servers.Williams and Berilla 2015, 92After these pages were populated, a final page was added to host a local Google “crawler” search box and listing of results. We limited this crawler to reveal only results located on the library’s digital archives pages and to conduct full-text searches of the converted PDF files.Wiedeman 2019b, 13Crawl limits document the boundaries of the crawler, including both the data or document limits for the crawl, and scoping rules that were applied.Lohndorf 2022Crawlers are software that identify materials on the live web that belong in your collections, based upon your choice of seeds and scope. A crawler refers to the automated agent, also called a robot or spider, that capture the live web content.
