harvester

(also spider), n.A software application designed to download pages from the Internet.

Notes

A harvester generally follows links on pages that it downloads, capturing the pages those links point to. A harvester often follows rules that limit its operations to a specific server or domain, the number of links to follow, and the kinds of files to download.