Package us.codecraft.webmagic.recover

Class MmapQueueScheduler

java.lang.Object

us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

us.codecraft.webmagic.recover.MmapQueueScheduler

All Implemented Interfaces:

Scheduler

public class MmapQueueScheduler
extends DuplicateRemovedScheduler

Author:

：linweisen

Field Summary

Fields inherited from class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

logger

Constructor Summary

Constructors

Constructor	Description
MmapQueueScheduler(DuplicateRemover duplicateRemover, java.lang.String path)

Method Summary

Modifier and Type	Method	Description
<T> T	fromJson(java.lang.String jsonString, java.lang.Class<T> clazz)
Request	poll(Task task)	get an url to crawl
void	pushWhenNoDuplicate(Request request, Task task)
java.lang.String	toJson(java.lang.Object object)

Methods inherited from class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

getDuplicateRemover, noNeedToRemoveDuplicate, push, setDuplicateRemover, shouldReserved

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MmapQueueScheduler

public MmapQueueScheduler(DuplicateRemover duplicateRemover,
                          java.lang.String path)

Method Detail

poll

public Request poll(Task task)

Description copied from interface: Scheduler

get an url to crawl

Parameters:

task - the task of spider

Returns:

the url to crawl

pushWhenNoDuplicate

public void pushWhenNoDuplicate(Request request,
                                Task task)

Overrides:

pushWhenNoDuplicate in class DuplicateRemovedScheduler

toJson

public java.lang.String toJson(java.lang.Object object)

fromJson

public <T> T fromJson(java.lang.String jsonString,
                      java.lang.Class<T> clazz)