Package us.codecraft.webmagic.scheduler

Class FileCacheQueueScheduler

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable, us.codecraft.webmagic.scheduler.MonitorableScheduler, us.codecraft.webmagic.scheduler.Scheduler
    public class FileCacheQueueScheduler
extends us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler
implements us.codecraft.webmagic.scheduler.MonitorableScheduler, java.io.Closeable
    Store urls and cursor in files so that a Spider can resume the status when shutdown.
    Since:
    0.2.0
    Author:
    code4crafter@gmail.com

    • Field Summary

      • Fields inherited from class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

        logger

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void close()  
      protected us.codecraft.webmagic.Request deserializeRequest​(java.lang.String line)  
      int getLeftRequestsCount​(us.codecraft.webmagic.Task task)  
      int getTotalRequestsCount​(us.codecraft.webmagic.Task task)  
      us.codecraft.webmagic.Request poll​(us.codecraft.webmagic.Task task)  
      protected void pushWhenNoDuplicate​(us.codecraft.webmagic.Request request, us.codecraft.webmagic.Task task)  
      protected java.lang.String serializeRequest​(us.codecraft.webmagic.Request request)  

      • Methods inherited from class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

        getDuplicateRemover, noNeedToRemoveDuplicate, push, setDuplicateRemover, shouldReserved

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      • Methods inherited from interface us.codecraft.webmagic.scheduler.Scheduler

        push

    • Constructor Detail

      • FileCacheQueueScheduler

        public FileCacheQueueScheduler​(java.lang.String filePath)

    • Method Detail

      • close

        public void close()
           throws java.io.IOException
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable
        Throws:
        java.io.IOException

      • pushWhenNoDuplicate

        protected void pushWhenNoDuplicate​(us.codecraft.webmagic.Request request,
                                   us.codecraft.webmagic.Task task)
        Overrides:
        pushWhenNoDuplicate in class us.codecraft.webmagic.scheduler.DuplicateRemovedScheduler

      • poll

        public us.codecraft.webmagic.Request poll​(us.codecraft.webmagic.Task task)
        Specified by:
        poll in interface us.codecraft.webmagic.scheduler.Scheduler

      • getLeftRequestsCount

        public int getLeftRequestsCount​(us.codecraft.webmagic.Task task)
        Specified by:
        getLeftRequestsCount in interface us.codecraft.webmagic.scheduler.MonitorableScheduler

      • getTotalRequestsCount

        public int getTotalRequestsCount​(us.codecraft.webmagic.Task task)
        Specified by:
        getTotalRequestsCount in interface us.codecraft.webmagic.scheduler.MonitorableScheduler

      • serializeRequest

        protected java.lang.String serializeRequest​(us.codecraft.webmagic.Request request)

      • deserializeRequest

        protected us.codecraft.webmagic.Request deserializeRequest​(java.lang.String line)