Custom HTTP clients

Copy for LLM

The Apify API client uses a pluggable HTTP client architecture. By default, it ships with an Impit-based HTTP client that handles retries, timeouts, passing headers, and more. You can replace it with your own implementation for use cases like custom logging, proxying, request modification, or integrating with a different HTTP library.

Default HTTP client

When you create an ApifyClient or ApifyClientAsync instance, it automatically uses the built-in ImpitHttpClient (or ImpitHttpClientAsync). This default client provides:

• Automatic retries with exponential backoff for network errors, HTTP 429, and HTTP 5xx responses.
• Configurable timeouts.
• Preparing request data and headers according to the API requirements, including authentication.
• Collecting requests statistics for monitoring and debugging.

You can configure the default client through the ApifyClient or ApifyClientAsync constructor:

Async client

from datetime import timedelta

from apify_client import ApifyClientAsync

TOKEN = 'MY-APIFY-TOKEN'


async def main() -> None:
    client = ApifyClientAsync(
        token=TOKEN,
        max_retries=4,
        min_delay_between_retries=timedelta(milliseconds=500),
        timeout=timedelta(seconds=360),
    )

Sync client

from datetime import timedelta

from apify_client import ApifyClient

TOKEN = 'MY-APIFY-TOKEN'


def main() -> None:
    client = ApifyClient(
        token=TOKEN,
        max_retries=4,
        min_delay_between_retries=timedelta(milliseconds=500),
        timeout=timedelta(seconds=360),
    )

Architecture

The HTTP client system is built on two key abstractions:

• HttpClient / HttpClientAsync - Abstract base classes that define the interface. Extend one of these to create a custom HTTP client by implementing the call method.
• HttpResponse - A runtime-checkable protocol that defines the expected response shape. Any object with the required attributes and methods satisfies the protocol — no inheritance needed.

To plug in your custom implementation, use the ApifyClient.with_custom_http_client class method.

All of these are available as top-level imports from the apify_client package:

from apify_client import (
    HttpClient,
    HttpClientAsync,
    HttpResponse,
)

The call method

The call method receives all the information needed to make an HTTP request:

• method - HTTP method (GET, POST, PUT, DELETE, etc.).
• url - Full URL to make the request to.
• headers - Additional headers to include.
• params - Query parameters to append to the URL.
• data - Raw request body (mutually exclusive with json).
• json - JSON-serializable request body (mutually exclusive with data).
• stream - Whether to stream the response body.
• timeout - Timeout for the request as a timedelta.

It must return an object satisfying the HttpResponse protocol.

The HTTP response protocol

HttpResponse is not a concrete class. Any object with the following attributes and methods will work:

Property / Method	Description
status_code: int	HTTP status code
text: str	Response body as text
content: bytes	Raw response body
headers: Mapping[str, str]	Response headers
json() -> Any	Parse body as JSON
read() -> bytes	Read entire response body
aread() -> bytes	Read entire response body (async)
close() -> None	Close the response
aclose() -> None	Close the response (async)
iter_bytes() -> Iterator[bytes]	Iterate body in chunks
aiter_bytes() -> AsyncIterator[bytes]	Iterate body in chunks (async)

note

Many HTTP libraries, including our default Impit or for example HTTPX already satisfy this protocol out of the box.

Plugging it in

Use the ApifyClient.with_custom_http_client (or ApifyClientAsync.with_custom_http_client) class method to create a client with your custom implementation:

Async client

from datetime import timedelta
from typing import Any

from apify_client import ApifyClientAsync, HttpClientAsync, HttpResponse

TOKEN = 'MY-APIFY-TOKEN'


class MyHttpClientAsync(HttpClientAsync):
    """Custom async HTTP client."""

    async def call(
        self,
        *,
        method: str,
        url: str,
        headers: dict[str, str] | None = None,
        params: dict[str, Any] | None = None,
        data: str | bytes | bytearray | None = None,
        json: Any = None,
        stream: bool | None = None,
        timeout: timedelta | None = None,
    ) -> HttpResponse: ...


async def main() -> None:
    client = ApifyClientAsync.with_custom_http_client(
        token=TOKEN,
        http_client=MyHttpClientAsync(),
    )

Sync client

from datetime import timedelta
from typing import Any

from apify_client import ApifyClient, HttpClient, HttpResponse

TOKEN = 'MY-APIFY-TOKEN'


class MyHttpClient(HttpClient):
    """Custom sync HTTP client."""

    def call(
        self,
        *,
        method: str,
        url: str,
        headers: dict[str, str] | None = None,
        params: dict[str, Any] | None = None,
        data: str | bytes | bytearray | None = None,
        json: Any = None,
        stream: bool | None = None,
        timeout: timedelta | None = None,
    ) -> HttpResponse: ...


def main() -> None:
    client = ApifyClient.with_custom_http_client(
        token=TOKEN,
        http_client=MyHttpClient(),
    )

After that, all API calls made through the client will go through your custom HTTP client.

warning

When using a custom HTTP client, you are responsible for constructing the request, handling retries, timeouts, and errors yourself. The default retry logic is not applied.

Use cases

Custom HTTP clients might be useful when you need to:

• Use a different HTTP library - Swap Impit for httpx, requests, or aiohttp.
• Route through a proxy - Add proxy support or request routing.
• Implement custom retry logic - Use different backoff strategies or retry conditions.
• Log requests and responses - Track API calls for debugging or auditing.
• Modify requests - Add custom fields, modify the body, or change headers.
• Collect custom metrics - Measure request latency, track error rates, or count API calls.

For a step-by-step walkthrough of building a custom HTTP client, see the Using HTTPX as the HTTP client guide.