ApifyHttpProxyMiddleware

apify.scrapy.middlewares.apify_proxy.ApifyHttpProxyMiddleware

Apify HTTP proxy middleware for Scrapy.

This middleware enhances request processing by adding a 'proxy' field to the request's meta and an authentication header. It draws inspiration from the HttpProxyMiddleware included by default in Scrapy projects. The proxy URL is sourced from the settings under the APIFY_PROXY_SETTINGS key. The value of this key, a dictionary, should be provided by the Actor input. An example of the proxy settings:

proxy_settings = {'useApifyProxy': true, 'apifyProxyGroups': []}

Index

Constructors

• __init__

Methods

• from_crawler
• process_exception
• process_request

Constructors

__init__

• __init__(proxy_settings): None

• Create a new instance.

  ---

  Parameters

  • proxy_settings: dict

  Returns None

Methods

from_crawler

• from_crawler(crawler): ApifyHttpProxyMiddleware

• Create an instance of ApifyHttpProxyMiddleware from a Scrapy Crawler.

  ---

  Parameters

  • crawler: Crawler

  Returns ApifyHttpProxyMiddleware

process_exception

• process_exception(request, exception, spider): None

• Process an exception that occurs during request processing.

  ---

  Parameters

  • request: Request

  • exception: Exception

  • spider: Spider

  Returns None

process_request

• async process_request(request, spider): None

• Process a Scrapy request by assigning a new proxy.

  ---

  Parameters

  • request: Request

  • spider: Spider

  Returns None