Package us.codecraft.webmagic.scheduler
Class RedisPriorityScheduler
- java.lang.Object
-
- us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler
-
- us.codecraft.webmagic.scheduler.RedisScheduler
-
- us.codecraft.webmagic.scheduler.RedisPriorityScheduler
-
- All Implemented Interfaces:
DuplicateRemover
,MonitorableScheduler
,Scheduler
public class RedisPriorityScheduler extends RedisScheduler
the redis scheduler with priority- Author:
- sai Created by sai on 16-5-27.
-
-
Field Summary
-
Fields inherited from class us.codecraft.webmagic.scheduler.RedisScheduler
pool
-
Fields inherited from class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler
logger
-
-
Constructor Summary
Constructors Constructor Description RedisPriorityScheduler(java.lang.String host)
RedisPriorityScheduler(redis.clients.jedis.JedisPool pool)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description Request
poll(Task task)
get an url to crawlprotected void
pushWhenNoDuplicate(Request request, Task task)
void
resetDuplicateCheck(Task task)
Reset duplicate check.-
Methods inherited from class us.codecraft.webmagic.scheduler.RedisScheduler
getItemKey, getLeftRequestsCount, getQueueKey, getSetKey, getTotalRequestsCount, isDuplicate
-
Methods inherited from class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler
getDuplicateRemover, noNeedToRemoveDuplicate, push, setDuplicateRemover, shouldReserved
-
-
-
-
Method Detail
-
pushWhenNoDuplicate
protected void pushWhenNoDuplicate(Request request, Task task)
- Overrides:
pushWhenNoDuplicate
in classRedisScheduler
-
poll
public Request poll(Task task)
Description copied from interface:Scheduler
get an url to crawl- Specified by:
poll
in interfaceScheduler
- Overrides:
poll
in classRedisScheduler
- Parameters:
task
- the task of spider- Returns:
- the url to crawl
-
resetDuplicateCheck
public void resetDuplicateCheck(Task task)
Description copied from interface:DuplicateRemover
Reset duplicate check.- Specified by:
resetDuplicateCheck
in interfaceDuplicateRemover
- Overrides:
resetDuplicateCheck
in classRedisScheduler
- Parameters:
task
- task
-
-