Skip to main content

ApifyHttpProxyMiddleware

apify.scrapy.middlewares.apify_proxy.ApifyHttpProxyMiddleware

Apify HTTP proxy middleware for Scrapy.

This middleware enhances request processing by adding a ‘proxy’ field to the request’s meta and an authentication header. It draws inspiration from the HttpProxyMiddleware included by default in Scrapy projects. The proxy URL is sourced from the settings under the APIFY_PROXY_SETTINGS key. The value of this key, a dictionary, should be provided by the Actor input. An example of the proxy settings:

proxy_settings = {‘useApifyProxy’: true, ‘apifyProxyGroups’: []}

Index

Constructors

Methods

Constructors

__init__

  • __init__(proxy_settings): None

  • Create a new instance.

    Parameters

    • proxy_settings: dict

    Returns None

Methods

from_crawler

  • Create an instance of ApifyHttpProxyMiddleware from a Scrapy Crawler.

    Parameters

    • crawler: Crawler

    Returns ApifyHttpProxyMiddleware

    Instance of the class.

process_exception

  • process_exception(request, exception, spider): None | Request

  • Process an exception that occurs during request processing.

    Parameters

    • request: Request
    • exception: Exception
    • spider: Spider

    Returns None | Request

process_request

  • async process_request(request, spider): None

  • Process a Scrapy request by assigning a new proxy.

    Parameters

    • request: Request
    • spider: Spider

    Returns None

    The request is processed and middleware pipeline can continue.