Uses of Class
us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler
-
Packages that use DuplicateRemovedScheduler Package Description us.codecraft.webmagic.recover us.codecraft.webmagic.samples.scheduler us.codecraft.webmagic.scheduler Scheduler is the part of url management. -
-
Uses of DuplicateRemovedScheduler in us.codecraft.webmagic.recover
Subclasses of DuplicateRemovedScheduler in us.codecraft.webmagic.recover Modifier and Type Class Description class
MmapQueueScheduler
-
Uses of DuplicateRemovedScheduler in us.codecraft.webmagic.samples.scheduler
Subclasses of DuplicateRemovedScheduler in us.codecraft.webmagic.samples.scheduler Modifier and Type Class Description class
DelayQueueScheduler
class
LevelLimitScheduler
-
Uses of DuplicateRemovedScheduler in us.codecraft.webmagic.scheduler
Subclasses of DuplicateRemovedScheduler in us.codecraft.webmagic.scheduler Modifier and Type Class Description class
FileCacheQueueScheduler
Store urls and cursor in files so that a Spider can resume the status when shutdown.class
PriorityScheduler
Priority scheduler.class
QueueScheduler
Basic Scheduler implementation.
Store urls to fetch in LinkedBlockingQueue and remove duplicate urls by HashMap.class
RedisPriorityScheduler
the redis scheduler with priorityclass
RedisScheduler
Use Redis as url scheduler for distributed crawlers.Methods in us.codecraft.webmagic.scheduler that return DuplicateRemovedScheduler Modifier and Type Method Description DuplicateRemovedScheduler
DuplicateRemovedScheduler. setDuplicateRemover(DuplicateRemover duplicatedRemover)
-