Package us.codecraft.webmagic.scheduler
Scheduler is the part of url management.
-
Interface Summary Interface Description MonitorableScheduler The scheduler whose requests can be counted for monitor.Scheduler Scheduler is the part of url management.
You can implement interface Scheduler to do: manage urls to fetch remove duplicate urls -
Class Summary Class Description DuplicateRemovedScheduler Remove duplicate urls and only push urls which are not duplicate.PriorityScheduler Priority scheduler.QueueScheduler Basic Scheduler implementation.
Store urls to fetch in LinkedBlockingQueue and remove duplicate urls by HashMap.