Package us.codecraft.webmagic.downloader
Interface Downloader
-
- All Known Implementing Classes:
AbstractDownloader
,HttpClientDownloader
,PhantomJSDownloader
,SeleniumDownloader
public interface Downloader
Downloader is the part that downloads web pages and store in Page object.
Downloader hassetThread(int)
method because downloader is always the bottleneck of a crawler, there are always some mechanisms such as pooling in downloader, and pool size is related to thread numbers.- Since:
- 0.1.0
- Author:
- code4crafter@gmail.com
-
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method Description Page
download(Request request, Task task)
Downloads web pages and store in Page object.void
setThread(int threadNum)
Tell the downloader how many threads the spider used.
-
-
-
Method Detail
-
download
Page download(Request request, Task task)
Downloads web pages and store in Page object.- Parameters:
request
- requesttask
- task- Returns:
- page
-
setThread
void setThread(int threadNum)
Tell the downloader how many threads the spider used.- Parameters:
threadNum
- number of threads
-
-