ApifyGracefulStopExtension

A Scrapy extension that stops the crawl gracefully when the Actor run is being aborted.

A graceful abort gives the run a moment before it is killed. The extension uses it to stop the engine: no new requests are started, the ones in flight finish along with their callbacks and item pipelines, and the scheduler marks them as handled in the request queue, so resurrecting the run does not download them again. A migration is handled by ApifyScheduler instead: the platform restarts a migrating run only if its process does not exit on its own, so the scheduler keeps the crawl running with nothing to do rather than stopping it.

Index

Methods

• __init__
• from_crawler
• spider_closed
• spider_opened

Methods

__init__

• __init__(crawler): None

• Parameters

  • crawler: Crawler

  Returns None

from_crawler

• from_crawler(crawler): ApifyGracefulStopExtension

• Create the extension and hook it up to the spider's lifecycle.

  ---

  Parameters

  • crawler: Crawler

  Returns ApifyGracefulStopExtension

spider_closed

• spider_closed(): None

• Stop listening for the abort of the Actor run.

  ---

  Returns None

spider_opened

• spider_opened(): None

• Start listening for the abort of the Actor run.

  ---

  Returns None