A program or automated script that browses the World Wide Web in a methodical, automated manner, typically for indexing web pages.