- All Known Implementing Classes:
public interface DownloaderDownloader is the part that downloads web pages and store in Page object.
setThread(int)method because downloader is always the bottleneck of a crawler, there are always some mechanisms such as pooling in downloader, and pool size is related to thread numbers.
downloadDownloads web pages and store in Page object.
void setThread(int threadNum)Tell the downloader how many threads the spider used.
threadNum- number of threads