Skip to main content

RequestQueue

Represents a queue storage for managing HTTP requests in web crawling operations.

The RequestQueue class handles a queue of HTTP requests, each identified by a unique URL, to facilitate structured web crawling. It supports both breadth-first and depth-first crawling strategies, allowing for recursive crawling starting from an initial set of URLs. Each URL in the queue is uniquely identified by a unique_key, which can be customized to allow the same URL to be added multiple times under different keys.

Data can be stored either locally or in the cloud. It depends on the setup of underlying storage client. By default a MemoryStorageClient is used, but it can be changed to a different one.

By default, data is stored using the following path structure:

{CRAWLEE_STORAGE_DIR}/request_queues/{QUEUE_ID}/{REQUEST_ID}.json
  • {CRAWLEE_STORAGE_DIR}: The root directory for all storage data specified by the environment variable.
  • {QUEUE_ID}: The identifier for the request queue, either "default" or as specified.
  • {REQUEST_ID}: The unique identifier for each request in the queue.

The RequestQueue supports both creating new queues and opening existing ones by id or name. Named queues persist indefinitely, while unnamed queues expire after 7 days unless specified otherwise. The queue supports mutable operations, allowing URLs to be added and removed as needed.

Usage

from crawlee.storages import RequestQueue

rq = await RequestQueue.open(name='my_rq')

Hierarchy

Index

Methods

Properties

Methods

__init__

  • __init__(id, name, storage_client): None

  • Parameters

    Returns None

add_request

  • Add a single request to the manager and store it in underlying resource client.

    Parameters

    • request: str | Request

      The request object (or its string representation) to be added to the manager.

    • optionalkeyword-onlyforefront: bool = False

      Determines whether the request should be added to the beginning (if True) or the end (if False) of the manager.

    Returns ProcessedRequest

add_requests_batched

  • async add_requests_batched(requests, *, batch_size, wait_time_between_batches, wait_for_all_requests_to_be_added, wait_for_all_requests_to_be_added_timeout): None

  • Add requests to the manager in batches.

    Parameters

    • requests: Sequence[str | Request]

      Requests to enqueue.

    • optionalkeyword-onlybatch_size: int = 1000

      The number of requests to add in one batch.

    • optionalkeyword-onlywait_time_between_batches: timedelta = timedelta(seconds=1)

      Time to wait between adding batches.

    • optionalkeyword-onlywait_for_all_requests_to_be_added: bool = False

      If True, wait for all requests to be added before returning.

    • optionalkeyword-onlywait_for_all_requests_to_be_added_timeout: timedelta | None = None

      Timeout for waiting for all requests to be added.

    Returns None

drop

  • async drop(): None

  • Drop the storage, removing it from the underlying storage client and clearing the cache.

    Returns None

fetch_next_request

  • async fetch_next_request(): Request | None

  • Return the next request in the queue to be processed.

    Once you successfully finish processing of the request, you need to call RequestQueue.mark_request_as_handled to mark the request as handled in the queue. If there was some error in processing the request, call RequestQueue.reclaim_request instead, so that the queue will give the request to some other consumer in another call to the fetch_next_request method.

    Note that the None return value does not mean the queue processing finished, it means there are currently no pending requests. To check whether all requests in queue were finished, use RequestQueue.is_finished instead.

    Returns Request | None

from_storage_object

  • from_storage_object(storage_client, storage_object): RequestQueue

get_handled_count

  • async get_handled_count(): int

get_info

  • Get an object containing general information about the request queue.

    Returns RequestQueueMetadata | None

get_request

  • async get_request(request_id): Request | None

  • Retrieve a request from the queue.

    Parameters

    • request_id: str

      ID of the request to retrieve.

    Returns Request | None

get_total_count

  • async get_total_count(): int

  • Return an offline approximation of the total number of requests in the source (i.e. pending + handled).

    Returns int

is_empty

  • async is_empty(): bool

is_finished

  • async is_finished(): bool

  • Check whether the queue is finished.

    Due to the nature of distributed storage used by the queue, the function might occasionally return a false negative, but it will never return a false positive.

    Returns bool

mark_request_as_handled

open

  • async open(*, id, name, configuration, storage_client): Storage

  • Open a storage, either restore existing or create a new one.

    Parameters

    • optionalkeyword-onlyid: str | None = None

      The storage ID.

    • optionalkeyword-onlyname: str | None = None

      The storage name.

    • optionalkeyword-onlyconfiguration: Configuration | None = None

      Configuration object used during the storage creation or restoration process.

    • optionalkeyword-onlystorage_client: StorageClient | None = None

      Underlying storage client to use. If not provided, the default global storage client from the service locator will be used.

    Returns Storage

reclaim_request

  • Reclaim a failed request back to the queue.

    The request will be returned for processing later again by another call to RequestQueue.fetch_next_request.

    Parameters

    • request: Request

      The request to return to the queue.

    • optionalkeyword-onlyforefront: bool = False

      Whether to add the request to the head or the end of the queue.

    Returns ProcessedRequest | None

storage_object

  • storage_object(storage_object): None

to_tandem

  • Combine the loader with a request manager to support adding and reclaiming requests.

    Parameters

    • optionalrequest_manager: RequestManager | None = None

      Request manager to combine the loader with. If None is given, the default request queue is used.

    Returns RequestManagerTandem

Properties

id

id: str

Get the storage ID.

name

name: str | None

Get the storage name.

storage_object

storage_object: StorageMetadata

Get the full storage object.

Page Options