Skip to main content

ApifyHttpProxyMiddleware

Apify HTTP proxy middleware for Scrapy.

This middleware enhances request processing by adding a 'proxy' field to the request's meta and an authentication header. It draws inspiration from the HttpProxyMiddleware included by default in Scrapy projects. The proxy URL is sourced from the settings under the APIFY_PROXY_SETTINGS key. The value of this key, a dictionary, should be provided by the Actor input. An example of the proxy settings:

proxy_settings = {'useApifyProxy': true, 'apifyProxyGroups': []}

Index

Methods

Methods

__init__

  • __init__(*, proxy_settings): None

  • Create a new instance.

    Parameters

    • optionalkeyword-onlyproxy_settings: dict

      Dictionary containing proxy settings, provided by the Actor input.

    Returns None

from_crawler

  • Create an instance of ApifyHttpProxyMiddleware from a Scrapy Crawler.

    Parameters

    • optionalkeyword-onlycrawler: Crawler

      Scrapy Crawler object.

    Returns ApifyHttpProxyMiddleware

process_exception

  • process_exception(*, request, exception, spider): None

  • Process an exception that occurs during request processing.

    Parameters

    • optionalkeyword-onlyrequest: Request

      Scrapy Request object.

    • optionalkeyword-onlyexception: Exception

      Exception object.

    • optionalkeyword-onlyspider: Spider

      Scrapy Spider object.

    Returns None

process_request

  • async process_request(*, request, spider): None

  • Process a Scrapy request by assigning a new proxy.

    Parameters

    • optionalkeyword-onlyrequest: Request

      Scrapy Request object.

    • optionalkeyword-onlyspider: Spider

      Scrapy Spider object.

    Returns None

Page Options