Skip to main content
Version: 3.1

Use HTTPX as the HTTP client

Copy for LLM

This guide shows how to replace the default ImpitHttpClient and ImpitHttpClientAsync with one based on HTTPX. The same approach works for any HTTP library — see Custom HTTP clients for the underlying architecture.

Why HTTPX?

You might want to use HTTPX instead of the default Impit-based client for reasons like:

  • You already use HTTPX in your project and want a single HTTP stack.
  • You need HTTPX-specific features.
  • You want fine-grained control over connection pooling or proxy routing.

Implementation

The implementation involves two steps:

  1. Extend HttpClient (sync) or HttpClientAsync (async) and implement the call method that delegates to HTTPX.
  2. Pass it to ApifyClient.with_custom_http_client to create a client that uses your implementation.

The call method receives parameters like method, url, headers, params, data, json, stream, and timeout. Map them to the corresponding HTTPX arguments — most map directly, except data which becomes HTTPX's content parameter and timeout which needs conversion from timedelta to seconds.

A convenient property of HTTPX is that its httpx.Response object already satisfies the HttpResponse protocol, so you can return it directly without wrapping.

One part of the contract isn't visible in the method signature: call must raise ApifyApiError for error responses instead of returning them. The resource clients rely on that error to work correctly. For example, get methods translate a 404 raised this way into a None return value, and user code handling ApifyApiError keeps working.

from __future__ import annotations

import asyncio
from http import HTTPStatus
from typing import TYPE_CHECKING, Any

import httpx

from apify_client import ApifyClientAsync
from apify_client.errors import ApifyApiError
from apify_client.http_clients import HttpClientAsync, HttpResponse

if TYPE_CHECKING:
    from apify_client.types import Timeout

TOKEN = 'MY-APIFY-TOKEN'


class HttpxClientAsync(HttpClientAsync):
    """Custom async HTTP client using HTTPX library."""

    def __init__(self) -> None:
        super().__init__()
        self._client = httpx.AsyncClient()

    async def call(
        self,
        *,
        method: str,
        url: str,
        headers: dict[str, str] | None = None,
        params: dict[str, Any] | None = None,
        data: str | bytes | bytearray | None = None,
        json: Any = None,
        stream: bool | None = None,
        timeout: Timeout = 'medium',
    ) -> HttpResponse:
        timeout_secs = self._compute_timeout(timeout, attempt=1) or 0

        # Merge the client's default headers (including authorization)
        # with the per-request ones.
        headers = self._merge_headers(self._headers, headers)

        response = await self._client.request(
            method=method,
            url=url,
            headers=headers,
            params=params,
            content=data,
            json=json,
            timeout=timeout_secs,
        )

        # Raising `ApifyApiError` for error responses is part of the `call`
        # contract. The resource clients rely on it, e.g. to translate a 404
        # into a `None` return value of `get` methods.
        if response.status_code >= HTTPStatus.BAD_REQUEST:
            raise ApifyApiError(response, attempt=1, method=method)

        # httpx.Response satisfies the HttpResponse protocol,
        # so it can be returned directly.
        return response


async def main() -> None:
    client = ApifyClientAsync.with_custom_http_client(
        token=TOKEN,
        http_client=HttpxClientAsync(),
    )

    actor = await client.actor('apify/hello-world').get()
    print(actor)


if __name__ == '__main__':
    asyncio.run(main())
warning

When using a custom HTTP client, you are responsible for handling retries, timeouts, and error handling yourself. The built-in retry logic with exponential backoff is part of the default ImpitHttpClient and is not applied to custom implementations.

Going further

The example above is minimal on purpose. In a production setup, you might want to extend it with:

  • Retry logic - Use HTTPX's event hooks or utilize library like tenacity to retry failed requests.
  • Custom headers - You can add headers in the call method before delegating to HTTPX.
  • Connection lifecycle - Close the underlying httpx.Client when done by adding a close() method to your custom client.
  • Proxy support - You can pass proxy=... when creating the httpx.Client.
  • Metrics collection - Track request latency, error rates, or other metrics by adding instrumentation in the call method.
  • Logging - Log requests and responses for debugging or auditing purposes.